Simple regression suite to test FPGA vs P8X32A
prof_braino
Posts: 4,313
A thread asked about testing the whole prop. http://forums.parallax.com/showthread.php/156887-A-good-coverage-software-test-suite-for-the-whole-Propeller1
also about exercising individual instructions http://forums.parallax.com/showthread.php/154461-Is-there-a-testbench-for-opcodes-%28-instruction-set-exersizer%29
My suggesttion is to consider the build and test automation Sal created. Starting with spin, it builds simple funtionality, and gradually adds optimization and develper functions to produce an assembler optimized executable for the prop, which happens to be the propforth PF5.5 kernel set. Since the build exercises every forth forth function, if it works and the resultant kernel produces the same kernel again, then nothing got broke. When changes are planned, all we need do is look for differences in results of the output log files, and we can determine if changes are good (and become our new target result) or bad (broke something) and need further investigation.
The build and test automation also exercises a significant protion of the prop instruction set. If the automation gives the same result on an FPGA version, then the FPGA might be equivalent to the P8X32A.
The automation is designed to encourage users to add custom tests to the suite. If we have additional prop opcodes or instructions we want to test, we just write a test, and stick on the end of the suite. One might test every opcode in the prop tthis way.
This can be used as a black box. One does not need to program using propforth to use the automation as a regression test. Checking for errors is just a "diff" between this result and the previous (target) result. Just look for any tests flagged as "different" and you can go off on whatever direction your verilog code goes.
Sal finds this a reasonable approach. He's going to trim off the superfluous kernels and tests, and organize for ease of adding user tests.
Ideally there will be a simple, explicite set of instructions for setting up and configuring the automation, and a minimal set of command to launch the suite.
Anyone that has tried the build and test automation for PF5.5 should be able to do the same. All the functionality exists and is included in the current download, but its organized for propforth kernel development rather than black box regression suite. It might be easier to wait for Sal to do some re-org and packaging.
If anybody is interested in using this method, I will post when we have package created.
also about exercising individual instructions http://forums.parallax.com/showthread.php/154461-Is-there-a-testbench-for-opcodes-%28-instruction-set-exersizer%29
My suggesttion is to consider the build and test automation Sal created. Starting with spin, it builds simple funtionality, and gradually adds optimization and develper functions to produce an assembler optimized executable for the prop, which happens to be the propforth PF5.5 kernel set. Since the build exercises every forth forth function, if it works and the resultant kernel produces the same kernel again, then nothing got broke. When changes are planned, all we need do is look for differences in results of the output log files, and we can determine if changes are good (and become our new target result) or bad (broke something) and need further investigation.
The build and test automation also exercises a significant protion of the prop instruction set. If the automation gives the same result on an FPGA version, then the FPGA might be equivalent to the P8X32A.
The automation is designed to encourage users to add custom tests to the suite. If we have additional prop opcodes or instructions we want to test, we just write a test, and stick on the end of the suite. One might test every opcode in the prop tthis way.
This can be used as a black box. One does not need to program using propforth to use the automation as a regression test. Checking for errors is just a "diff" between this result and the previous (target) result. Just look for any tests flagged as "different" and you can go off on whatever direction your verilog code goes.
Sal finds this a reasonable approach. He's going to trim off the superfluous kernels and tests, and organize for ease of adding user tests.
Ideally there will be a simple, explicite set of instructions for setting up and configuring the automation, and a minimal set of command to launch the suite.
Anyone that has tried the build and test automation for PF5.5 should be able to do the same. All the functionality exists and is included in the current download, but its organized for propforth kernel development rather than black box regression suite. It might be easier to wait for Sal to do some re-org and packaging.
If anybody is interested in using this method, I will post when we have package created.
Comments
I just reorganized the Github repo that Heater started and Mindrobots expanded, so that it only contains one tree of files for the Altera targets (I'm a beginner with VHDL/AHDL/Verilog and I feel dangerous enough to do this with the Quartus sources; I don't have any Xilinx hardware so I didn't do any testing on the Pipistrello tree).
I expect to be adding more features in the future, and a good test suite that runs the P1V through the wringer would really be beneficial, especially if it can be automated somehow so I can run a script to build various combinations of different versions of the tree with different features, and run the tests against each combination (or at least some of them).
===Jac
I have been busy travelling and other things for a while so I had no time to take care of that repo.
It would be great if you made a pull request for your changes and I could look them over and do a quick test on them. If they fly on my nano I can merge them.
Chip has been, and is busy, with other things as well but did contact me again regarding taking over my repo for Parallax recently. So I hope we can do that soon.
jac: Nice work. If I get a link a can check the Xilinx-tree. I don't have the actual pipistrello-hw but I can at least build it.
I'm just about to add a new xilinx-target (as you know) so this would be beneficial to NOT having the same files in lots of places.
Overclocked: Magnus Karlsson's Github fork is at this link.
Thanks!
===Jac