Shop OBEX P1 Docs P2 Docs Learn Events
P2 Test Program Plans?? (Now's the time, while we are waiting) — Parallax Forums

P2 Test Program Plans?? (Now's the time, while we are waiting)

mindrobotsmindrobots Posts: 6,506
edited 2014-03-04 22:20 in Propeller 2
Testing Board Inventory (last update 3/3/14):

ozpropdev (Brian) - DE0 + adapter, DE2 + adapter
potatohead (Doug) - DE2
Sapieha - DE0, DE2
David Betz - DE0, DE2 + adapter
Bill Henning - DE0 + adapter, 2xDE0 w/o adapter, DE2 + adapter
mindrobots (Rick) - 2xDE0 w/o adapter, DE2 + adapter when back in stock
cluso99 - DE0 + adapter
jazzed - DE2 + adapter
tubular - DE0 + adapter, DE2 + adapter
Roy Eltham - DE2 + adapter (when back in stock)

TESTING GITHUB REPOSITORY - grab a GitHub ID if you don't have one and join the fun!


******** YOU CAN PRETTY MUCH IGNORE THE RAMBLINGS BELOW - SKIP TO POST #2 FOR CHIP'S PLAN *********

I'm thinking about the day when there is a "final" candidate for the FPGA and it comes time to testing. I (and many others that are playing along) have never done anything like this before. It seems that there are many experts who have been contributing to the design over the past several months that have. Now's the time to lend more of your expertise and experience in design, testing and fabrication to help get the test program off to a successful start.

I have bunches of questions but few answers.

I assume since Chip is the sole developer and most valuable resource, the testing program needs to be structured and managed by others with experience to keep his time free for addressing real issues and continuing development on Spin2 and other things that only he can do right now. With that in mind:
  • How do we document and track discovered issues? (this will be important to avoid duplication of efforts and reporting) It will take people that are experienced and can say, "this looks like the same item reported in #12".
  • Can the more experienced programmers and designers help to vett issues to make sure they are real potential problem and not either coding issues or misunderstanding of how things are supposed to work?
  • What do we need to actually test effectively at various levels? Oscilloscope? Logic Analyzer? ??
  • Are there established procedure how to go about methodically testing a new architecture?
  • Are there hardware or software test jigs or templates that someone needs to come up with?
  • Are there "corner cases" or places where the smart folks think problems might be lurking (timing issues, complexity issues, etc.)?
  • What can we test that actually test things that are going from the FPGA to the Silicon (versus FPGA glue that won't appear on the real chip.)
  • What can't we test because it isn't viable until the real chip is out? Obviously thermal, some timing issues, voltage issues and such other physical characteristics need the real chip but what other things in our test FPGA aren't part of the real chip?
  • Can the two items above be put into a checklist/matrix so everyone one has an idea what to test, if it's been tested, results, etc.
  • Who wants to step forward and be the test manager(s) - one or two of the people with experience in this.

Am I making too big a deal of the testing phase? Will it work, be productive and valuable and make a difference if people just run off in their own directions to try things that they have a vested interest in or think are interesting features? I'm pretty sure we can't just say, "Go forth and test an we'll meet back here in a month or two to see if it's good. Don't forget to report anything strange you see."

It sounds like some of you have done this before, please provide feedback, now is another time to draw on your expertise!

Thoughts, comments, tell me to just go away?

Ready to test either with a plan and some guidance or off finding things that look interesting to me.
«1

Comments

  • cgraceycgracey Posts: 14,206
    edited 2014-02-28 06:18
    mindrobots wrote: »
    I'm thinking about the day when there is a "final" candidate for the FPGA and it comes time to testing. I (and many others that are playing along) have never done anything like this before. It seems that there are many experts who have been contributing to the design over the past several months that have. Now's the time to lend more of your expertise and experience in design, testing and fabrication to help get the test program off to a successful start.

    I have bunches of questions but few answers.

    I assume since Chip is the sole developer and most valuable resource, the testing program needs to be structured and managed by others with experience to keep his time free for addressing real issues and continuing development on Spin2 and other things that only he can do right now. With that in mind:
    • How do we document and track discovered issues? (this will be important to avoid duplication of efforts and reporting) It will take people that are experienced and can say, "this looks like the same item reported in #12".
    • Can the more experienced programmers and designers help to vett issues to make sure they are real potential problem and not either coding issues or misunderstanding of how things are supposed to work?
    • What do we need to actually test effectively at various levels? Oscilloscope? Logic Analyzer? ??
    • Are there established procedure how to go about methodically testing a new architecture?
    • Are there hardware or software test jigs or templates that someone needs to come up with?
    • Are there "corner cases" or places where the smart folks think problems might be lurking (timing issues, complexity issues, etc.)?
    • What can we test that actually test things that are going from the FPGA to the Silicon (versus FPGA glue that won't appear on the real chip.)
    • What can't we test because it isn't viable until the real chip is out? Obviously thermal, some timing issues, voltage issues and such other physical characteristics need the real chip but what other things in our test FPGA aren't part of the real chip?
    • Can the two items above be put into a checklist/matrix so everyone one has an idea what to test, if it's been tested, results, etc.
    • Who wants to step forward and be the test manager(s) - one or two of the people with experience in this.

    Am I making too big a deal of the testing phase? Will it work, be productive and valuable and make a difference if people just run off in their own directions to try things that they have a vested interest in or think are interesting features? I'm pretty sure we can't just say, "Go forth and test an we'll meet back here in a month or two to see if it's good. Don't forget to report anything strange you see."

    It sounds like some of you have done this before, please provide feedback, now is another time to draw on your expertise!

    Thoughts, comments, tell me to just go away?

    Ready to test either with a plan and some guidance or off finding things that look interesting to me.


    Testing is a really tedious endeavor that's sometimes even boring. What I figure we can do, that doesn't stress anyone that much, is have me do a bunch of reality checks, to make sure there's nothing grossly wrong, and then have you guys do whatever you want with it and report any funny behavior. Nothing tests things like people just using them.

    At some point, I will post my test case code and you guys can augment it in whatever ways you want. It's a simple framework that just needs an LED to work.
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-02-28 06:33
    cgracey wrote: »
    Testing is a really tedious endeavor that's sometimes even boring. What I figure we can do, that doesn't stress anyone that much, is have me do a bunch of reality checks, to make sure there's nothing grossly wrong, and then have you guys do whatever you want with it and report any funny behavior. Nothing tests things like people just using them.

    At some point, I will post my test case code and you guys can augment it in whatever ways you want. It's a simple framework that just needs an LED to work.

    Cool! I got an LED!

    Party on! :lol:
  • potatoheadpotatohead Posts: 10,261
    edited 2014-02-28 07:34
    IMHO, having the test code will tell us a lot. Each of us will see cases that we can go deep on, or will due to our interests and skills.

    That recent COGNEW bug got found due to me being lazy, and I'm pretty sure that was not on the radar, and for a while I thought it was me and being unfamiliar with the dynamics. It manifested on an earlier build and I didn't say anything, because a little code shuffling saw it go away. Thought it was me. It wasn't. We need to be conscious of that dynamic.

    Chip is right. We need people doing things most of all. And we need to be humble, talk about it and discover when it's broken as opposed to us not understanding something fully, etc...

    I've a friend who authors tests for a fairly complex web application for a software company you all probably know. Great people, awesome product, complex. A proposed test for this? Years he says to get it done. A real test program will have been written as features were made. That's Chip's code, and from there user testing carries the day. I don't think we have a bad model here. We just need to do it.

    Perhaps we can get at this another way. It might be good to understand what has seen some use and how. This, coupled with test code already in use (Chip, and our programs), will get us to areas that need some exercise. If we have that discussion, it's easier for some of us to pick something, go learn it, then test it, IMHO.
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-02-28 07:51
    That's why I started this thread. Ask and learn.
    I don't know and others do! :0)
  • Bill HenningBill Henning Posts: 6,445
    edited 2014-02-28 07:56
    Your friend is absolutely right.

    A "proper" formal test plan would take an insane amount of time - probably between 3 - 6 man-months to write.
    potatohead wrote: »
    IMHO, having the test code will tell us a lot. Each of us will see cases that we can go deep on, or will due to our interests and skills.

    That recent COGNEW bug got found due to me being lazy, and I'm pretty sure that was not on the radar, and for a while I thought it was me and being unfamiliar with the dynamics. It manifested on an earlier build and I didn't say anything, because a little code shuffling saw it go away. Thought it was me. It wasn't. We need to be conscious of that dynamic.

    Chip is right. We need people doing things most of all. And we need to be humble, talk about it and discover when it's broken as opposed to us not understanding something fully, etc...

    I've a friend who authors tests for a fairly complex web application for a software company you all probably know. Great people, awesome product, complex. A proposed test for this? Years he says to get it done. A real test program will have been written as features were made. That's Chip's code, and from there user testing carries the day. I don't think we have a bad model here. We just need to do it.

    Perhaps we can get at this another way. It might be good to understand what has seen some use and how. This, coupled with test code already in use (Chip, and our programs), will get us to areas that need some exercise. If we have that discussion, it's easier for some of us to pick something, go learn it, then test it, IMHO.
  • Dave HeinDave Hein Posts: 6,347
    edited 2014-02-28 08:11
    Bill Henning wrote: »
    A "proper" formal test plan would take an insane amount of time - probably between 3 - 6 man-months to write.
    Seems like that might would fit into the latest P2 schedule. Now we just need a test-plan guy to organize the task. Prof_Braino, are you watching this thread? :)
  • Ym2413aYm2413a Posts: 630
    edited 2014-02-28 08:47
    I think the best bet would be to just start a thread on here in the P2 area of the forum.

    Keep the bug reports off of the main P2 development post. That thread has gotten so large, things get lost in there. : ]
    I hope to jump on to testing once the FPGA image is finalized.
  • ozpropdevozpropdev Posts: 2,793
    edited 2014-02-28 16:59
    Hi All
    What would be useful to establish is how many of us with FPGA boards are out there in testland? :)
    Brian = DE0-Nano + DE2-115 + adapter boards
  • potatoheadpotatohead Posts: 10,261
    edited 2014-02-28 17:11
    I have DE2
  • SapiehaSapieha Posts: 2,964
    edited 2014-02-28 17:36
    Sapieha = DE0-Nano + DE2-115+ some others FPGA's to test my programing
  • David BetzDavid Betz Posts: 14,516
    edited 2014-02-28 17:43
    DE2-115 + adapter board
    DE0-Nano without an adapter board
  • Bill HenningBill Henning Posts: 6,445
    edited 2014-02-28 18:17
    1x DE2-115 + adapter
    1x DE0-Nano + adapter
    2x DE0-Nano (without adapter)
    ozpropdev wrote: »
    Hi All
    What would be useful to establish is how many of us with FPGA boards are out there in testland? :)
    Brian = DE0-Nano + DE2-115 + adapter boards
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-02-28 18:50
    I'll update the first post with a list, when i get to a real computer, so we have an inventory in one place.

    2 x DE0 Nano w/o adapter
    1 x DE2 w/ adapter (when they come in stock)

    1 x DE0 adapter up for adoption
  • Cluso99Cluso99 Posts: 18,069
    edited 2014-03-01 02:38
    DE0 + adapter

    My plans...
    1. P2 LMM Debugger - convert to hubexec mode
    2. FS USB - continue with 2 new instructions when available
  • potatoheadpotatohead Posts: 10,261
    edited 2014-03-01 07:51
    From the discussion last night, I think it's worth at least documenting the instructions that have been in programs. We won't know all the cases for those instructions without doing that ugly unit test program, but we can see what has seen a little exercise as opposed to nothing or a lot. That's one sort of "have we tested it at all?" map all of us will find useful. Can anyone else suggest some other basics, like perhaps common things to test on?

    I'm thinking we should test on those things we would put in the end user documentation to explain what an instruction does. Take a math instruction as an example, what happens at zero, overflow, negative, etc... For something like the RDxxxx WRxxxx instructions, we probably should test for bogus input, and edge cases like address wrapping, etc... And we would be prepping the data needed for those docs too. A twofer!

    We won't get 'em all that way, but we would get the ones people are highly likely to use, plus a few others that happen as we code things to explore. And we get a lot, quick!
  • Bill HenningBill Henning Posts: 6,445
    edited 2014-03-01 08:07
    I think we need a Wiki for testing, so results do not get buried in a thread

    We need to test

    - every possible source/destination addressing mode on a sample instruction
    - every instruction ideally with every addressing mode, but as long as the addressing modes are tested separately, idealism can wait :)
    - prefetch
    - code caching
    - data caching
    - counters
    - video helpers
    - xfer
    - aux
    - lifo
    - uarts
    - SERDES
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-03-01 08:59
    Multi-tasking lots to test there before we even get to multi-thread. Cog only, cog and hi exec mixed, all the unique registers, starting, stopping, slice mixes, etc.

    I've been using SERIN/OUT but only single instance on the Nano.

    I've been playing with single aux stacks, tried two - one PTRX and one PTRY, either have a coding bug (most likely) or a P2 bug. I need to isolate it and test more.

    That's about all I've done with any structure.

    I can try to set up a wiki or maybe a Google code sit then we can have documents, download and even a code repo for the test suites? Whatever I do won't happen until tonight so leave ideas until then.
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-03-01 09:03
    PS - If anyone wants to be a moderator/manager/admin on the wiki or whatever, send me a PM with your email (I think they usually can do that with email addresses)
  • potatoheadpotatohead Posts: 10,261
    edited 2014-03-01 09:09
    doug dot dingus at gmail dot com

    I did this with the propeller wiki, and it's good to have a couple people "on file" in case somebody goes away, etc... We might as well get started!
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-03-01 09:15
    I actually now yours! One of the few people in my google circles or whatever they are called.

    Pure wiki or google code site?
  • potatoheadpotatohead Posts: 10,261
    edited 2014-03-01 09:29
    Cool Yeah, I've not done too much with that, but I am following a fair number of people. Saw you in there a few times. All good.
  • jazzedjazzed Posts: 11,803
    edited 2014-03-01 11:36
    I have a DE2 + Adapter.

    I'm plenty busy already, so I'm waiting for the GCC P2 compiler ... which is waiting for the instruction set to stop changing :)
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-03-01 17:20
    I started a GitHub repository for P2-FPGA-Testing

    You need to have a GitHub membership to play along.

    Let me know your GitHub ID an I can add you as a collaborator.

    It's pretty empty now with just a readme file. I'll try and add some files to the repository...if nothing else just so I can practice with Git and different file types.

    Let the games begin!!
  • potatoheadpotatohead Posts: 10,261
    edited 2014-03-01 17:23
    Github?

    I'm thinking we will have test code, docs, issues, etc... worth a look. I can't right now, but I know a ton of projects end up there. Heater would recommend it. :)
  • David BetzDavid Betz Posts: 14,516
    edited 2014-03-01 17:29
    potatohead wrote: »
    Github?

    I'm thinking we will have test code, docs, issues, etc... worth a look. I can't right now, but I know a ton of projects end up there. Heater would recommend it. :)
    I don't think github provides a downloads area either. I have several projects there and I don't remember that facility.
  • mindrobotsmindrobots Posts: 6,506
    edited 2014-03-01 17:32
    potatohead wrote: »
    Github?

    I'm thinking we will have test code, docs, issues, etc... worth a look. I can't right now, but I know a ton of projects end up there. Heater would recommend it. :)

    I was looking there. The problem with Github and most repository hosting services is the files are "bundled up" into a project and you check out the entire project and work with that collection of files. I'll keep looking.

    If anyone knows of a place that supports:
    Documentation (wiki)
    File Downloads - individual files
    Issue tracking
    Repository support

    shout out!
  • Heater.Heater. Posts: 21,230
    edited 2014-03-01 18:24
    mindrobots,
    ...shout out!
    GITHUB!
    You can download individual files from github. Using it's "raw" feature. It's easy.

    For example go here:https://github.com/ZiCog/fftbench

    Let's say you want a copy of just fftbench.c. OK click on that.
    At the top of the listing of the source is a row of buttons. Click on the "Raw" button.
    Now it's just a right click and "save file as".

    Github of course supports a wiki for documentation and has an issue tracker.

    Now, actually what is the problem with using a whole repository anyway? If people are wanting to contribute shouldn't they be doing that. Once you have cloned a repository once a "pull" will only fetch updates. Very quick and easy. Why fight it?
  • David BetzDavid Betz Posts: 14,516
    edited 2014-03-01 18:27
    Heater. wrote: »
    mindrobots,

    GITHUB!
    You can download individual files from github. Using it's "raw" feature. It's easy.

    For example go here:https://github.com/ZiCog/fftbench

    Let's say you want a copy of just fftbench.c. OK click on that.
    At the top of the listing of the source is a row of buttons. Click on the "Raw" button.
    Now it's just a right click and "save file as".

    Github of course supports a wiki for documentation and has an issue tracker.

    Now, actually what is the problem with using a whole repository anyway? If people are wanting to contribute shouldn't they be doing that. Once you have cloned a repository once a "pull" will only fetch updates. Very quick and easy. Why fight it?
    Does this "raw mode" work with binary files like the FPGA configuration files?
  • Heater.Heater. Posts: 21,230
    edited 2014-03-01 18:47
    Yep.

    I just added an fftbench executable to the above repo. When you click on it the content is not displayed but if you hit the "Raw" button the thing is downloaded. For some reason a ".txt" extension is added but it is the correct binary file.

    Out of curiosity I added the same file a "fftbench.bin". That gets downloaded with the name unchanged.
  • David BetzDavid Betz Posts: 14,516
    edited 2014-03-01 18:48
    Heater. wrote: »
    Yep.

    I just added an fftbench executable to the above repo. When you click on it the content is not displayed but if you hit the "Raw" button the thing is downloaded. For some reason a ".txt" extension is added but it is the correct binary file.

    Out of curiosity I added the same file a "fftbench.bin". That gets downloaded with the name unchanged.
    Nice! I didn't know about that feature.
Sign In or Register to comment.