Shop OBEX P1 Docs P2 Docs Learn Events
My attempt at a "Gold Standard" process - Page 6 — Parallax Forums

My attempt at a "Gold Standard" process

123468

Comments

  • SRLMSRLM Posts: 5,045
    edited 2013-01-08 12:45

    Well, no, that's not a very good reference either. The better Wikipedia page would be the UART page (http://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter), but it's still missing implementation details that would allow for testing.

    I doubt that there will be a good external reference for UART, since it seems to be mostly an industry consensus rather than standard like I2C. Of course, nothing says that we can't come up with our own "AppNote" and test the objects against that.
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-08 13:04
    mindrobots wrote: »
    This isn't even a statement of what this code does. This is a statement of what we want/expect this code to do.

    And how we intend to demonstrate that it does that, in a way that anyone can demonstrate the identical result for them-self, and get the identical result.

    The order of you list might be re-arranged, this is how I see it:
    1. Item is either crowd-source nominated to be Goldified or self-nominated
    2. We need an <insert language here> object to do this function according to this specification, data sheet or other set of requirements.
    3. - here are the references we intend to use.
    4. - here are the tests we indent to perform
    5. - here are the results we expect to see
    6. Here is the coding standard for <insert language here> we intend to follow
    7. //SQE Questions and review process - as each supporting item is complete
    8. {...writing code is left as an exercise for someone...}
    9. Author completes the code and enters as "Wood" in OBEX
    10. Here is the code in <insert language here> that should perform this function according to this specification. Test and demo wrappers are provided.
    11. - All tests are performed on the code, and the results are analyzed
    12. iterate process with author and reviewers until all tests pass, no requirements change, no tests change, and no code changes; [ if until team loses interest work simply stops]
    13. object for CANDIDATE for GOLDEN, this decision is left to parallax or the community at large to evaluate if the result has any particular value

    And the final step, when the item is out in the wild:

    Folks use the item, and identify that tests do not properly meet the existing requirements, or request changes to the requirements (and thus trigger modifications to tests and code)
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-08 13:19
    The important thing is if #8 coding happened AFTER steps 1-7 (getting out ducks in a row), it is very easy to get the code right, it can do exactly what we want because we asked for it up front. It is very difficult to make mistakes that get out of development, bugs generally do not make it out into the wild.

    If coding happens before ANY of 1-7, the likelihood increases that it does not do something we want, since we have not asked for it yet, and these result in bugs in the field.

    Non conformance to requirements is the definition of a bug. Doing 1-7 first prevents bugs, doing 8 first creates bugs.
  • jazzedjazzed Posts: 11,803
    edited 2013-01-08 16:45
    Generally that's looking much more useful. Thanks for investing time to make the list. Does it outline the requirements for finishing your 90% ? If so, great. If not, add to it.

    Nevertheless, I hope this is a starting point for everyone. It is for me although I reserve the right to change my mind on some details as I see fit.

    I'm assuming the "future-tense" tone of the list/goals means that any existing modules to be updated will have lots of reverse engineering and design work done up front before someone proceeds with "goldification." I guess people will accept that if they are willing to do the work for new or existing modules.

    The idea that 1-7 necessarily preceding 8 is already blown of course for the existing Parallax objects because the code already exists. One can do 1-7 and then check the existing code.

    BTW, not all unit tests and functions/methods can be predicted in advance for private functions. I assume your item 4 is only for requirements of public APIs.

    Also, everything is "Wood" right now ....

    How many casual programmers are willing to go through the process?
    I am depending on what Parallax wants.

    And how we intend to demonstrate that it does that, in a way that anyone can demonstrate the identical result for them-self, and get the identical result.

    The order of you list might be re-arranged, this is how I see it:
    1. Item is either crowd-source nominated to be Goldified or self-nominated
    2. We need an <insert language here> object to do this function according to this specification, data sheet or other set of requirements.
    3. - here are the references we intend to use.
    4. - here are the tests we indent to perform
    5. - here are the results we expect to see
    6. Here is the coding standard for <insert language here> we intend to follow
    7. //SQE Questions and review process - as each supporting item is complete
    8. {...writing code is left as an exercise for someone...}
    9. Author completes the code and enters as "Wood" in OBEX
    10. Here is the code in <insert language here> that should perform this function according to this specification. Test and demo wrappers are provided.
    11. - All tests are performed on the code, and the results are analyzed
    12. iterate process with author and reviewers until all tests pass, no requirements change, no tests change, and no code changes; [ if until team loses interest work simply stops]
    13. object for CANDIDATE for GOLDEN, this decision is left to parallax or the community at large to evaluate if the result has any particular value
    And the final step, when the item is out in the wild:

    Folks use the item, and identify that tests do not properly meet the existing requirements, or request changes to the requirements (and thus trigger modifications to tests and code)
  • CircuitsoftCircuitsoft Posts: 1,166
    edited 2013-01-08 16:53
    I suspect that the process will require more practice to get used to it, than work on each particular object. I'm trying to Practice with the ffds1 driver, but the initial requirement of finding a spec for it to follow is proving challenging. At this point it appears to be de-facto, and that rs232-type devices are interoperable only because of "tribal knowledge". A co-worker of mine who has read the EIA-232 standard has derided it for having no useful detail. Apparently almost anything will match the spec.
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-08 19:31
    jazzed wrote: »
    Does it outline the requirements for finishing your 90% ?

    The requirements, tests, code, testing, results analysis, and review of intermediate products at each step of the way is the 100% I would strive for.

    I reserve the right to change my mind on some details as I see fit.

    Anyone can change or modify any set of these steps at any time, as long as they do the courtesy of letting us know whats being done or not done. Then we can look at why those parts are skipped. Maybe some things can be taken out. Maybe the guy needs extra help. But if we know, we can respond.
    The idea that 1-7 necessarily preceding 8 is already blown of course for the existing Parallax objects because the code already exists.

    This is the really cool part. As far as the "process" is concerned, the subsequent items are not "present" until the precedent items are in place. Any existing code is just a draft of "test code" or "proof of concept". Once the requirements are spelled out, and the tests are getting worked out, we tend to find that the code needs some fixin', so the existing code will get versioned up anyway. Each fix represents a bug in the field that has been squashed before anyone (outside of development) is troubled. We can count these, and determine the frequency and severity of each, and get hard data as to whether the process is a waste of time or totally awesome, or something in between.
  • jazzedjazzed Posts: 11,803
    edited 2013-01-08 19:43
    The requirements, tests, code, testing, results analysis, and review of intermediate products at each step of the way is the 100% I would strive for.

    Great. I consider my request from this post much further on it's way to being closed then.
    It's a shame about 100 posts have passed since then. Now we can be less wasteful.

    All we have to do is:

    * fill in the details
    * make sure the process works
    * see if the process is acceptable to everyone

    Thanks,
    --Steve
  • Brian FairchildBrian Fairchild Posts: 549
    edited 2013-01-08 23:40
    Maybe use TI 16C450 datasheet, page 11 as a reference?
    The 16450 and 16550 are pretty much THE de-facto industry standard devices, more so the 16550. So why not pick that as your reference device? The 16550 is also available as an IP unit from a number of places, who simply reference a particular manufacturers device, and so is documented both as dedicated silicon and as VHDL/Verilog source. At least one implementation is available on opencores.org.
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-09 06:12
    SRLM wrote: »
    Well, no, that's not a very good reference either. The better Wikipedia page would be the UART page (http://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter), but it's still missing implementation details that would allow for testing.

    I doubt that there will be a good external reference for UeART, since it seems to be mostly an industry consensus rather than standard like I2C. Of course, nothing says that we can't come up with our own "AppNote" and test the objects against that.

    Do we expect to be required to implement and test things like "break condition" this point?
    Also, at this point we DON'T want to specify implementation detail until we are clear on what we are looking at. We want the "what" not "how".

    On the prop, serial communication can be done in software, and doesn't need any additional hardware (unless its needed to talk to some particular device). I recommend doing something that does not require additional hardware, this is the more basic case and easier to duplicate and test.
  • ctwardellctwardell Posts: 1,716
    edited 2013-01-09 06:25
  • mindrobotsmindrobots Posts: 6,506
    edited 2013-01-09 06:45

    On the prop, serial communication can be done in software, and doesn't need any additional hardware (unless its needed to talk to some particular device). I recommend doing something that does not require additional hardware, this is the more basic case and easier to duplicate and test.

    While not needing additional hardware or testing/verification is a good thing, running COG to COG on a Prop with the software under test on both COGs isn't a valid test scenario. For example, if your bit timing is off or is slipping, there's a good chance it could be slipping in the same way on both COGs since it's the same object. You need to test against an accepted standard - SPI device for SPI, I2C for I2C and something external and async/uart-ish for this case. A common accepted terminal program on a PC for example (minicom, TeraTerm, propeller terminal, etc.) would be a valid test/verification case.
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-09 06:56
    jazzed wrote: »
    It's a shame about 100 posts have passed since then. Now we can be less wasteful.

    No, offense, but this is the other big misconception, and a very common mistake. We can't skip the discussion.

    We either have the discussion upfront, during planning, when it will prevent bugs and do some good; or we have it later, when we have to deal with all the bugs that got out to the field. Fixing a bug from the field costs about 100x the cost of fixing it in development, particularly it the bug is correctable in the planning phase. (if anyone doesn't believe this or wants to discuss it further we can make a separate thread).

    This discussion is the most efficient way to get everybody on the same page. Every minute we spend working things out now save ten minutes to an hour of arguments and recriminations over a bug returning from the field.

    Another rule of thumb is "Doing it fast takes ten times longer then simply doing it".
    This is the same rule as "Measure twice, cut once; measure once, cut yourself".
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-09 07:07
    mindrobots wrote: »
    ... running COG to COG on a Prop with the software under test on both COGs isn't a valid test scenario.... A common accepted terminal program on a PC for example (minicom, TeraTerm, propeller terminal, etc.) would be a valid test/verification case.

    COG to COG can get us most of the way, and makes it inexpensive and simplefor anyone to easily implement and reproduce. Connecting the result to minicon or teraterm on a PC can verify the last bit. In fact, this is the way serial on the prop is always tested on another project, and has never had an issue. Of course this might be due to the skill of the programmer, but its what I've observed.

    This will also help subdivide the functionality, since it encourages keeping the hardware support separate from the serial protocol support. Having a two tier test might be the way to go.
  • prof_brainoprof_braino Posts: 4,313
    edited 2013-01-09 08:51
    We haven't reached concensus on reference to define serial, we want to address this so we don't stall.

    We want to say: "This is the set of function we need to in order to do serial communication." We want the MINIMUM necessary and sufficient to have something we can call "serial communication". We can limit this to Asynchronous serial, or we can include Synchronous. We can limit the intial pass to a software only implementation on the prop, or we can include external hardware UART. We want to do whatever is easiest to move forward.

    Circuit soft, Please select the one or ones you can use. You are doing all the work, you get to choose. If we have any issues with your selections, we will let you know, and you decide if you want to act on any suggestion of not.

    At this point, we just want to get started, if we find we made a wrong selection we can always revisit the topic when we have better information.

    So far we have this list of options (if I missed some they are still in the running)

    From circuitsoft:
    http://en.wikipedia.org/wiki/Asynchronous_serial_communication
    http://www.ti.com/lit/ds/symlink/tl16c450.pdf

    from the community:
    http://www.atmel.com/Images/AVR274.pdf
    http://micromouse.sdsu.edu/uart/microUart.pdf
    http://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter
    http://www.unm.edu/~zbaker/ece238/slides/UART.pdf

    I will make a suggestion since this is the first pass at the process. Normally I have minimal participation in the decision making process, I just note that the conversation happened and agreement was reached.
    My thought is start with:
    http://en.wikipedia.org/wiki/Asynchronous_serial_communication
    and
    http://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter
    Start with prop only software implementation, and add external hardware after.

    Agree/Disagree?
  • mindrobotsmindrobots Posts: 6,506
    edited 2013-01-09 09:02
    Start with prop only software implementation, and add external hardware after.

    Probably just not understanding but I'm having trouble swallowing this. If I'm going to sit down to develop this in an incremental fashion (place favorite buzzword here), I don't want my initial testing to be against the same unknown I'm developing. I want a known working example on one side of the equation. When done, I have a working new implementation that is tested against a working established implementation. If my test goal is to test prop to prop, but I NEED to code against a working something, then my test environment doesn't match my development environment....

    So I'll ask it THIS TIME, what am I missing?
  • Heater.Heater. Posts: 21,230
    edited 2013-01-09 09:36
    What is going on here?
    If the "gold standard" debated here were applied to the Prop II design the Prop II would not be out before the sun went supernova. At which point the "gold standard" process would still not be defined:)
  • mindrobotsmindrobots Posts: 6,506
    edited 2013-01-09 10:01
    Heater. wrote: »
    before the sun went supernova.

    Wow, something like that could REALLY mess with delivery dates! :frown:

    Can we build that into the test plan?
  • jazzedjazzed Posts: 11,803
    edited 2013-01-09 10:25
    Heater. wrote: »
    What is going on here?
    If the "gold standard" debated here were applied to the Prop II design the Prop II would not be out before the sun went supernova. At which point the "gold standard" process would still not be defined:)

    LOL

    Actually we have already been part of at least one supernova. Before that though more matter in our galactic neighborhood was just air ....

    We have just started converting air into tangible measures of progress. Still a lot of fluff floating around though, and I don't expect that will ever be transformed.
  • mindrobotsmindrobots Posts: 6,506
    edited 2013-01-09 10:41
    jazzed wrote: »

    We have just started converting air into tangible measures of progress.

    Cold air, being more dense, seems easier to convert.....we seem to have problems with converting hot air! :lol:
  • jazzedjazzed Posts: 11,803
    edited 2013-01-09 11:33
    mindrobots wrote: »
    Cold air, being more dense, seems easier to convert.....we seem to have problems with converting hot air! :lol:

    I was going to mention the physics of that. Having cold air is necessary to mitigate the effects of gravity on a stellar nursery for example.


    Where does the friction (heat) come from?

    In space it's gravity (and other forces to a point); here, it's mainly communication problems.

    The forum communications methodology seems to make the friction much worse than necessary. Being able read in detail and take things at face value rather than reading what is not written would help tremendously. In a face to face situation or even on the phone, it is much easier to understand people without using much brain power.

    That being said, to extend the space analogy, It's clear to me that if you want something done in a reasonable amount of time instead of 14 billion years, you must have a design (this is not intended to be some religion -vs- science thing, so don't take it that way). The only way to succeed with a design is to have clear requirements and be able clearly communicate goals so that all participants can follow along and be productive.
  • rod1963rod1963 Posts: 752
    edited 2013-01-09 11:45
    I don't think Jobs and Wozniak would have ever created the Apple if they went about it this way. The same goes for Ford and his Model T.

    Even when I spent time doing IT for AF flight test programs, I didn't see this level of complexity among coders. They wrote, tested and documented the code. Yes, the programmers actually documented their own code! There were no "process" people involved unless they were managers, who wisely kept their noses out of the day to day work of their programmers.
  • mindrobotsmindrobots Posts: 6,506
    edited 2013-01-09 11:48
    jazzed wrote: »
    The only way to succeed with a design is to have clear requirements and be able clearly communicate goals so that all participants can follow along and be productive.

    There's a proper time to blow smoke and a proper time to shoot arrows (be direct and on target, not pointed and hurtful). Knowing when to do which is critical to success (or survival).
  • jazzedjazzed Posts: 11,803
    edited 2013-01-09 11:55
    When only two people in a garage are creating something the process is very simple. When you get other people involved (external manufacturing for example), you must be able to specify the product.

    I have a program in mind. Can you please write it for me?
  • jazzedjazzed Posts: 11,803
    edited 2013-01-09 12:02
    mindrobots wrote: »
    There's a proper time to blow smoke and a proper time to shoot arrows (be direct and on target, not pointed and hurtful). Knowing when to do which is critical to success (or survival).

    Sure. Being able to understand when something is not working and knowing to change course is also important. We all have problems with that at times, and it is made much worse by talking past one another rather than being direct.
  • Heater.Heater. Posts: 21,230
    edited 2013-01-09 12:11
    rod1963,

    Interesting. I have been on the other end of this "process". For two years I was part of the team testing the Primary Flight Computer software for the Boeing 777. You know, the fly by wire stuff.

    The design, or should I say requirements, document came from Boeing and when my boss printed it out was a stack of paper over a meter and a half high.

    There were thirty or so guys turning that requirement into a software/hardware design and writing the code. There was another thirty or so guys analysing the same requirements and creating test software. A parallel activity. Designers, coders, reviewers, testers, for any given component were not allowed to be the same people.

    That's before we talk about the "process" people who make sure that everyone is playng their part in this correctly.

    I have been involved in similar processes in many other aerospace and miltary projects.

    On the other hand...

    I belive the original Full Duplex Serial object was created by Chip. It was thrown over the fence and has worked very well ever since. Mean while he has moved on to designing the PII, thank God, and we are still here debating the process required to get the perfect FDS object.

    Hence my, seemingly flippant, question, what is going on here?
  • CircuitsoftCircuitsoft Posts: 1,166
    edited 2013-01-09 12:35
    Heater. wrote: »
    I believe the original Full Duplex Serial object was created by Chip. It was thrown over the fence...
    and has been shown to be problematic in some cases. Lonesock appears to have fixed it (and my serial driver is a port of his to C), but we still need a standard to test it against.

    The best option would be a mixed-signal oscilloscope with serial decoding and statistics, but that's expensive ($10k+).
  • Heater.Heater. Posts: 21,230
    edited 2013-01-09 12:45
    Circuitsoft,

    Yes, one guy had a problem with extreme timing requirements in one application.

    Does that mean we should spend forever trying to spec and create the perfect FDS?

    As you say Lonesock fixed it anyway.
  • CircuitsoftCircuitsoft Posts: 1,166
    edited 2013-01-09 12:48
    Lonesock appears to have fixed, it, so I want to find the documentation that shows that, and pull it all into one place. Also, it's possible that I broke it, so I want to verify what I've done as well.

    I have also had problems with Chip's original FDS, (With the SparkFun Serial LCD display) but haven't gone much further to test things out now (need to find said display).
  • Heater.Heater. Posts: 21,230
    edited 2013-01-09 13:02
    Circuitsoft,

    No worries, you obviously have a vested interest in making sure that there is an FDS that functions as well as it can.
    That is natural and seems to be the way the open source world works, effort gets expended where someone needs it. The results of their efforts go back into the pool.

    However if we apply this process idea to all objects in the OBEX, for example, we are going to be there forever. And, well, most people won't know or don't care.

    Perhaps I'm wrong and I don't get the idea, I don't know.
  • jazzedjazzed Posts: 11,803
    edited 2013-01-09 14:09
    Heater. wrote: »
    ....
    Perhaps I'm wrong and I don't get the idea, I don't know.

    Maybe your right Heater. OBEX is a free-for-all that allows a free exchange of effort. This Gold Standard is supposed to be a professional thing. Everyone's expectations are different. Professional has different meanings to different people depending on their perspective.

    IMHO, opening all this up to the community to finish 6 or more months ago has been a mistake so far. Maybe Parallax will have time to achieve their goals later in a more reasonable way. I thought they had a great start with their document, and maybe they can finish.

    I just believe that if someone needs to judge something that there should be ratified objective criteria - I'll play by that.

    I wish this was fun again.
Sign In or Register to comment.