byte 1: LSB [0-5] chan 1 low [6] 0b0 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
byte 2: LSB [0-5] chan 1 high [6] 0b0 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
byte 1: LSB [0-5] chan 2 low [6] 0b1 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
byte 2: LSB [0-5] chan 2 high [6] 0b1 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
That way, the Javascript can figure out, after reading 2 or 3 bytes where in the packet it is (especially ay the beginning of the stream)
One more thing - the first byte sent should be an ASCII 13. (Carriage return) There is some connection info that gets sent before the data stream, so there has to be a character to delineate that.
byte 1: Low byte of channel 1
byte 2: [0-3] high bits for channel 1, [4-6] Channel for this and the previous byte, [7] Unused?
Repeat these two bytes for each channel currently enabled
To tweak your suggestion slightly:
bit [7] could mark the first byte in a burst.
That's pretty much what the Coding I have above already does, only it has a bit per byte to preserve framing, so you know what is Hi/low data.
That bit pair is alternating 0,1,0,1,0,1,0,1 for data streams, but that allows 1,1 and 0,0 for 'mark the first byte in a burst' stuff.
The ADC payload bit ordering is largely dictated by the ADC, as the focus is on best Baud speeds, for continual streaming. That bit-order can be re-shuffled at the PC end, in a few lines.
the base64 thing happens between the BlocklyProp-client and the browser. More exactly, USB data comes into the BlocklyProp-client RAW, then gets base64 encoded, and sent to the browser via a Websocket. When the browser gets it, it is then base64 unencoded. This is because certain ASCII chars would break the Websocket. When we move to an offline version, we won't need the Websocket because the client and browser will be within the same app, so at that point the base64 encoding goes away. Unfortunately, there isn't really a way to conditionally use encoding or not - it's sort of an all or nothing thing. That being said - some speed tests will prove out what's possible.
Hopefully, the data can be captured via the USB driver handler, before the base64 is forced on it.
Both - I was mistaken in thinking the packets should be timestamped - if the prop sends at regular intervals, that should be sufficient. So I like David's simple packet suggestion.
I do suspect that we will want to Prop to manage the trigger...but again, open to what's possible, and letting testing prove out what's best.
Yes, strict streaming has implicit exact time base, crystal precision - for 3MBaud on a 80MHz Prop, that gives 6700.00ns data result streaming repeats.
Trigger is really just a matter of pausing the start, so the tight stream code is not affected. A small polling loop that does no UART Tx, but reads/checks ADC for _/= or =\_ (or either?) is enough
I'm now looking at ways to improve edge timing on the same stream. - for Digital not analog use.
I think careful use of Timer modes (POS detector), and some PC-side inference, can let the same 6700.00ns data rate, resolve edges on 2 channels, to an easy 100ns/channel (0..67 counts change/chan)
A simple 7/4 scale maybe can improve that to 17.5MHz average (a mix of 50ns and 62.5ns LSB ) to pack a count change of 0..118 into the available 0..127 field
Uses :
A 1ms PWM servo pulse can be measured on this to 0.01%/100ppm easily, or 57ppm scaled. (or 50ppm if set for single channel only)
I think serial data up to 57600 could be capture & decoded on a system.
A GPS 1pps can report to 0.1ppm, giving a means to calibrate that Prop Crystal
Okay, did some testing - 921600 baud seems to be the upper limit. It's mostly garbage when it goes higher than that...
I'll post some stuff that anyone can try, hopefully by evening, if not by tomorrow.
The browser side stuff is here. Basically, I took a copy of the relevant stuff from BlockyProp and made a new Github repo for it here: https://github.com/MatzElectronics/BlocklyScope
Open and connect the BlocklyProp client application,
download and extract as a .zip, open index.html with Chrome or Firefox,
click the second blue circle button (graph) to open a graph.
The relevant code that receives data and graphs it is in cdn/blocklyc.js, starting at line 327
P.S. if chartist.js (the graphing engine) is too slow, we can use an HTML5 canvas instead (what PhiPi used in his SpinScope). Nothing wrong with that method at all - may probably be better anyway...I just haven't had time to dive too deep into it...
Baud rate means nothing at all if the throughput isn't there. If you try to run at 921600 with errors and your throughput is only 11,500 characters/second then 115.2k baud is all that you need. I know in the Spin objects that the TP was lousy, not sure about Blockly.
Baud rate means nothing at all if the throughput isn't there. If you try to run at 921600 with errors and your throughput is only 11,500 characters/second then 115.2k baud is all that you need. I know in the Spin objects that the TP was lousy, not sure about Blockly.
?
I take Matt's comment as meaning 921600 is actually ok and higher than that gave issues.
That's 8x better than 115200, but still ~1/3 of the thoughput a FT231X can manage
and maybe 10% of what compiles PC code can manage on Terminal tests. (Prop itself can manage 4MB as a SPI-UART bridge,)
eg I can get 8MBd sustained throughput thru an Exar UART(RX) FT2232H Tx, on a compiled terminal app - sending file.
I think I've done even 12MBd duplex echo on FT2232H before. So the PC HW limit is above that.
Matt: I wonder if a FT2232H nudges that topbaud rate up any higher ? - can you try that ?
Valid test increments on FT2232H should be 1M,1.0909', 1.2M, 1.333',1.5M, 1.714285M, 2M
I know that's not a Prop system UART, but it can be useful to push these problems from differing directions to see where the limits are being imposed.
I'm not sure where the chars drop, as it seems you can define the Baud and Buffer sizes on Open Port, and filling the buffer should be interrupt based within the drivers ?
Default buffer mentioned in one github as 64k, did you try something like 1M, & sending 300k blocks ? ( < Buffer size)
@jmg - It's that "fine" line between theory vs practice and if you did this for a living you would know this and wouldn't be using a question mark.
Transmitting from the Prop to a PC at 921600 is not the same as transmitting 92,160 bytes/second with one character every 10.85us. With 100% throughput you will achieve that but is that really the case? How long does the software loop take? Many Spin objects for instance are not capable of high TP even if they are capable of high baud rates.
An update on this - while I think we can do this, and I'm very interested in some help with the code on the Propeller side, we may have to wait until we get to the offline/app version of BlocklyProp The JavaScript is plenty fast - but right now it appears to be "chunking" the data - basically buffering a whole bunch of characters before passing them to the browser. So instead of a nice stream of characters - we are getting large chunks of data intermittently. I'm not entirely sure if there is anything we can do about it, or that we'd want to as changing that behavior might break other things.
One possible workaround is to wait for a chunk, and then knowing how far apart the samples are and how many samples there are, feed them to the graph/scope fluidly - basically it would look like a scope, but it would be delayed by however much time it took for the first chunk to arrive (maybe 100ms?) - this is pretty easy to do in JS.
Additionally, the Chartist.js engine doesn't seem well suited for this, so I've been taking a look at how PhiPi used an HTML5 canvas, and that looks much more promising. I'm working out some kinks now, but once I do, I'll update the Github repo with the HTML5 canvas code.
More to come, but DO keep at the code on the Propeller end - finding a good, efficient way to send two channels to the computer and manage things like a trigger would be great to have ready for this when I can work out the kinks on the computer end
jmg - I'm fairly confident that the issue isn't the hardware, and most likely it's within the browser's JS/serial object. Nearly everything that happens in browser is "sandboxed" so it's usually limited compared to what can be done closer to the silicon. Abstraction sucks, but that's the bounding box on this one
That's why my goals are modest - and geared for the classroom - enough scope to show a servo pulse, a low frequency audio wave, or a slow baud uart comm.
An update on this - while I think we can do this, and I'm very interested in some help with the code on the Propeller side, we may have to wait until we get to the offline/app version of BlocklyProp The JavaScript is plenty fast - but right now it appears to be "chunking" the data - basically buffering a whole bunch of characters before passing them to the browser. So instead of a nice stream of characters - we are getting large chunks of data intermittently. I'm not entirely sure if there is anything we can do about it, or that we'd want to as changing that behavior might break other things.
That's ok, and there will always be some chunking - that's why it's worth experimenting with setting a larger buffer, and also testing a faster UART. On the SW we've used, Buffer size can be passed to the windows Open serial code.
example from calls from FreeBasic is here http://bourabai.kz/einf/freebasic/KeyPgOpenCom.html
Buffers up to 1~3MBytes are ok on modern PCs
FT231X & CP2102N have 512 byte buffers in each direction, whilst the FT2232H has 4k buffers, as well as the faster Frame rate of HS-USB. That's why we find the FT2232H is a very good 'second opinion' test vehicle, on these real world shake-out tests
The key number is when increasing the PC Buffer size, no longer improves baud throughput.
We did all this when shaking out a terminal for maximal performance - we wanted to be sure the terminal SW was not the bottleneck. It helps uncovers little details around what you can change, and what you cannot.
With this code, you do have the luxury of not needing fast turn-around times, and you also know how many samples you are expecting per request from the Prop.
More to come, but DO keep at the code on the Propeller end - finding a good, efficient way to send two channels to the computer and manage things like a trigger would be great to have ready for this when I can work out the kinks on the computer end
I did think of a variant of this mode I had before
...careful use of Timer modes (POS detector), and some PC-side inference, can let the same 6700.00ns data rate, resolve edges on 2 channels, to an easy 100ns/channel (0..67 counts change/chan)
A simple 7/4 scale maybe can improve that to 17.5MHz average (a mix of 50ns and 62.5ns LSB ) to pack a count change of 0..118 into the available 0..127 field
That set of numbers assumes 3MBd and 7bits per report per timer, but if that nudges to be 10 bits per report, for 3 bytes content of 2 channels, (up from 2) you now have 30b slots for 100k reports/second, and the possible 10b/1024 change in Timer lower bits, can resolve down to a clock rate of 9.765625ns - ie it can capture pulse-widths of Servos, down to the Prop's timer resolution.
At 80MHz that's 12.5ns, but someone running 100MHz could get 10ns
All of this can scale down slower, but it seems a shame to lower the 'what's possible' due to PC side limitations.
Smarter to try to push those up, as you have already done..
eg if you cannot improve at all on the 921600 value, it could be possible to report 14b timer LSB captures (4 bytes for 2 channels)
The maths for that (let's use 100MHz) is 10n*2^14 => <= 163.84us possible frame delays
Time for 4 bytes ? = 40/921600 => frame delay of 43.403us, so that does fit ok (good margin).
A 43us report/sample update is fine for servos, & should be ok for most GPS 1pps signals, and should be ok to sample/report 9600 Baud data, with precise baud rate measurement, but not much above that baud speed.
( well, maybe with caveats... 9600 can time edges and decode random data, but if you limit higher bauds to known-simple-data, you could still confirm Baud Rate settings, and that is still lab/class room useful. eg send 0x00 at 115200, is 78.125us pulse width, which the streaming 43.403us frames can report.
I like the idea of precision resolve to the limits of the Prop's timers, even if the PC link is somewhat more modest.
The chunking is a problem I initially had with SpinScope. I solved it by setting a variable ($| = 1) in the Perl server that forces constant flushing to the output channel, rather than waiting for the buffer to fill.
basically it would look like a scope, but it would be delayed by however much time it took for the first chunk to arrive (maybe 100ms?) - this is pretty easy to do in JS
.
That delay is no big deal. Definitely decouple display stream rates from sample rates. Once it's captured, maybe averages of captures running, PC display is a different problem. A nice setup will work fast and clean on both. But a not so nice setup will likely still be very useful.
Having read through this whole thing, building it to work on lowest common denominator baud rate makes sense. Many classrooms have whatever they can find, or mooch.
Be sure and reference that guy. The students will better understand what their prop scope block can tell them.
A few Hz refresh is sufficient for a lot of basic tasks, IMHO. If they know it runs that way, and they have a faster button, most of the initial "will it work?" worries go away.
Whatever it shows should be accurate (as the sample data is), then simple, then fast, then advanced, again, IMHO.
Going that route means minimum viable block need not do much beyond an accurate, simple display. Once working, options, optimization will come and be welcomed, tested by users.
Above all, initial, if limited, success should be extremely robust. Just works. Works better, faster and the fiddling one may have to do, depending, flows much better when users are invested, getting something to build on.
eg if you cannot improve at all on the 921600 value, it could be possible to report 14b timer LSB captures (4 bytes for 2 channels)
The maths for that (let's use 100MHz) is 10n*2^14 => <= 163.84us possible frame delays
Time for 4 bytes ? = 40/921600 => frame delay of 43.403us, so that does fit ok (good margin).
A 43us report/sample update is fine for servos, & should be ok for most GPS 1pps signals, and should be ok to sample/report 9600 Baud data, with precise baud rate measurement, but not much above that baud speed.
( well, maybe with caveats... 9600 can time edges and decode random data, but if you limit higher bauds to known-simple-data, you could still confirm Baud Rate settings, and that is still lab/class room useful. eg send 0x00 at 115200, is 78.125us pulse width, which the streaming 43.403us frames can report.
I like the idea of precision resolve to the limits of the Prop's timers, even if the PC link is somewhat more modest.
Bumping this... I was unsure about the GPS 1pps pulse width, and info is sparse on many products, so I dug out a GPS from Adafruit and measured that.
On my counter, it reports a HI-time of 100.0019ms and a repeat of 1.0000029Hz (ie my counter timebase is off by a stable 2.9ppm)
Assuming other GPS units are similar, that makes the 43us report rate above, quite ok.
You could encode base64, into 5 bytes for a 30 bit report record, to pack 27 bits of timing, and one bit of level and 2 bits of channel ID Tag - that becomes rate agnostic down to an underflow limit of 1.342s (100MHz) or 1.6777216s (80MHz), - so that does capture GPS 1pps nicely, and has some upper limit based on report rates.
At 5 bytes & 921600 test limit, that's appx 55us 'shutter recovery time' - & that is ok to capture a 9600 Baud random UART character.
Or 6 bytes can pack a 32b payload, and 4 bits of payload ID, and still manage 9600 baud capture at 921600 peak burst.
You might find that short packets (here 5~6 bytes), sparsely sent, are ok at even higher speeds, which would drop that 'shutter recovery time' effect. (5 bytes/3MBs is ~17us send time.)
ie you rely on the hardware buffers, which are ~ 512 bytes on FT231X / CP2102N devices, to smooth.
Careful pairing of WAIT and POS detector / NEG detector hardware modes, could capture one channel down to quite narrow pulse widths (<< send time), and with granularity equal to SysCLK precision.
Comments
byte 1: LSB [0-5] chan 1 low [6] 0b0 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
byte 2: LSB [0-5] chan 1 high [6] 0b0 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
byte 1: LSB [0-5] chan 2 low [6] 0b1 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
byte 2: LSB [0-5] chan 2 high [6] 0b1 [7] 0b0 for most packets, 0b1 for FIRST packet in burst.
That way, the Javascript can figure out, after reading 2 or 3 bytes where in the packet it is (especially ay the beginning of the stream)
That's pretty much what the Coding I have above already does, only it has a bit per byte to preserve framing, so you know what is Hi/low data.
That bit pair is alternating 0,1,0,1,0,1,0,1 for data streams, but that allows 1,1 and 0,0 for 'mark the first byte in a burst' stuff.
The ADC payload bit ordering is largely dictated by the ADC, as the focus is on best Baud speeds, for continual streaming. That bit-order can be re-shuffled at the PC end, in a few lines.
Hopefully, the data can be captured via the USB driver handler, before the base64 is forced on it.
Yes, strict streaming has implicit exact time base, crystal precision - for 3MBaud on a 80MHz Prop, that gives 6700.00ns data result streaming repeats.
Trigger is really just a matter of pausing the start, so the tight stream code is not affected. A small polling loop that does no UART Tx, but reads/checks ADC for _/= or =\_ (or either?) is enough
I'm now looking at ways to improve edge timing on the same stream. - for Digital not analog use.
I think careful use of Timer modes (POS detector), and some PC-side inference, can let the same 6700.00ns data rate, resolve edges on 2 channels, to an easy 100ns/channel (0..67 counts change/chan)
A simple 7/4 scale maybe can improve that to 17.5MHz average (a mix of 50ns and 62.5ns LSB ) to pack a count change of 0..118 into the available 0..127 field
Uses :
A 1ms PWM servo pulse can be measured on this to 0.01%/100ppm easily, or 57ppm scaled. (or 50ppm if set for single channel only)
I think serial data up to 57600 could be capture & decoded on a system.
A GPS 1pps can report to 0.1ppm, giving a means to calibrate that Prop Crystal
I'll post some stuff that anyone can try, hopefully by evening, if not by tomorrow.
that seems quite modest -guess it shows software really Is sloowing down faster than HW is speeding up
The browser side stuff is here. Basically, I took a copy of the relevant stuff from BlockyProp and made a new Github repo for it here: https://github.com/MatzElectronics/BlocklyScope
Open and connect the BlocklyProp client application,
download and extract as a .zip, open index.html with Chrome or Firefox,
click the second blue circle button (graph) to open a graph.
The relevant code that receives data and graphs it is in cdn/blocklyc.js, starting at line 327
Here is some initial PropC code just for testing - please please do what you will with it
demo.blockly.parallax.com/blockly/editor/blocklyc.jsp?project=1226
If we can make something work with the above starting points, then we are on to something!
?
I take Matt's comment as meaning 921600 is actually ok and higher than that gave issues.
That's 8x better than 115200, but still ~1/3 of the thoughput a FT231X can manage
and maybe 10% of what compiles PC code can manage on Terminal tests. (Prop itself can manage 4MB as a SPI-UART bridge,)
eg I can get 8MBd sustained throughput thru an Exar UART(RX) FT2232H Tx, on a compiled terminal app - sending file.
I think I've done even 12MBd duplex echo on FT2232H before. So the PC HW limit is above that.
Matt: I wonder if a FT2232H nudges that topbaud rate up any higher ? - can you try that ?
Valid test increments on FT2232H should be 1M,1.0909', 1.2M, 1.333',1.5M, 1.714285M, 2M
I know that's not a Prop system UART, but it can be useful to push these problems from differing directions to see where the limits are being imposed.
I'm not sure where the chars drop, as it seems you can define the Baud and Buffer sizes on Open Port, and filling the buffer should be interrupt based within the drivers ?
Default buffer mentioned in one github as 64k, did you try something like 1M, & sending 300k blocks ? ( < Buffer size)
Transmitting from the Prop to a PC at 921600 is not the same as transmitting 92,160 bytes/second with one character every 10.85us. With 100% throughput you will achieve that but is that really the case? How long does the software loop take? Many Spin objects for instance are not capable of high TP even if they are capable of high baud rates.
One possible workaround is to wait for a chunk, and then knowing how far apart the samples are and how many samples there are, feed them to the graph/scope fluidly - basically it would look like a scope, but it would be delayed by however much time it took for the first chunk to arrive (maybe 100ms?) - this is pretty easy to do in JS.
Additionally, the Chartist.js engine doesn't seem well suited for this, so I've been taking a look at how PhiPi used an HTML5 canvas, and that looks much more promising. I'm working out some kinks now, but once I do, I'll update the Github repo with the HTML5 canvas code.
More to come, but DO keep at the code on the Propeller end - finding a good, efficient way to send two channels to the computer and manage things like a trigger would be great to have ready for this when I can work out the kinks on the computer end
That's why my goals are modest - and geared for the classroom - enough scope to show a servo pulse, a low frequency audio wave, or a slow baud uart comm.
example from calls from FreeBasic is here http://bourabai.kz/einf/freebasic/KeyPgOpenCom.html
Buffers up to 1~3MBytes are ok on modern PCs
FT231X & CP2102N have 512 byte buffers in each direction, whilst the FT2232H has 4k buffers, as well as the faster Frame rate of HS-USB. That's why we find the FT2232H is a very good 'second opinion' test vehicle, on these real world shake-out tests
The key number is when increasing the PC Buffer size, no longer improves baud throughput.
We did all this when shaking out a terminal for maximal performance - we wanted to be sure the terminal SW was not the bottleneck. It helps uncovers little details around what you can change, and what you cannot.
With this code, you do have the luxury of not needing fast turn-around times, and you also know how many samples you are expecting per request from the Prop.
I did think of a variant of this mode I had before
That set of numbers assumes 3MBd and 7bits per report per timer, but if that nudges to be 10 bits per report, for 3 bytes content of 2 channels, (up from 2) you now have 30b slots for 100k reports/second, and the possible 10b/1024 change in Timer lower bits, can resolve down to a clock rate of 9.765625ns - ie it can capture pulse-widths of Servos, down to the Prop's timer resolution.
At 80MHz that's 12.5ns, but someone running 100MHz could get 10ns
All of this can scale down slower, but it seems a shame to lower the 'what's possible' due to PC side limitations.
Smarter to try to push those up, as you have already done..
eg if you cannot improve at all on the 921600 value, it could be possible to report 14b timer LSB captures (4 bytes for 2 channels)
The maths for that (let's use 100MHz) is 10n*2^14 => <= 163.84us possible frame delays
Time for 4 bytes ? = 40/921600 => frame delay of 43.403us, so that does fit ok (good margin).
A 43us report/sample update is fine for servos, & should be ok for most GPS 1pps signals, and should be ok to sample/report 9600 Baud data, with precise baud rate measurement, but not much above that baud speed.
( well, maybe with caveats... 9600 can time edges and decode random data, but if you limit higher bauds to known-simple-data, you could still confirm Baud Rate settings, and that is still lab/class room useful. eg send 0x00 at 115200, is 78.125us pulse width, which the streaming 43.403us frames can report.
I like the idea of precision resolve to the limits of the Prop's timers, even if the PC link is somewhat more modest.
-Phil
That delay is no big deal. Definitely decouple display stream rates from sample rates. Once it's captured, maybe averages of captures running, PC display is a different problem. A nice setup will work fast and clean on both. But a not so nice setup will likely still be very useful.
Having read through this whole thing, building it to work on lowest common denominator baud rate makes sense. Many classrooms have whatever they can find, or mooch.
Be sure and reference that guy. The students will better understand what their prop scope block can tell them.
A few Hz refresh is sufficient for a lot of basic tasks, IMHO. If they know it runs that way, and they have a faster button, most of the initial "will it work?" worries go away.
Whatever it shows should be accurate (as the sample data is), then simple, then fast, then advanced, again, IMHO.
Going that route means minimum viable block need not do much beyond an accurate, simple display. Once working, options, optimization will come and be welcomed, tested by users.
Above all, initial, if limited, success should be extremely robust. Just works. Works better, faster and the fiddling one may have to do, depending, flows much better when users are invested, getting something to build on.
My .02, worth what you paid.
Bumping this... I was unsure about the GPS 1pps pulse width, and info is sparse on many products, so I dug out a GPS from Adafruit and measured that.
On my counter, it reports a HI-time of 100.0019ms and a repeat of 1.0000029Hz (ie my counter timebase is off by a stable 2.9ppm)
Assuming other GPS units are similar, that makes the 43us report rate above, quite ok.
You could encode base64, into 5 bytes for a 30 bit report record, to pack 27 bits of timing, and one bit of level and 2 bits of channel ID Tag - that becomes rate agnostic down to an underflow limit of 1.342s (100MHz) or 1.6777216s (80MHz), - so that does capture GPS 1pps nicely, and has some upper limit based on report rates.
At 5 bytes & 921600 test limit, that's appx 55us 'shutter recovery time' - & that is ok to capture a 9600 Baud random UART character.
Or 6 bytes can pack a 32b payload, and 4 bits of payload ID, and still manage 9600 baud capture at 921600 peak burst.
You might find that short packets (here 5~6 bytes), sparsely sent, are ok at even higher speeds, which would drop that 'shutter recovery time' effect. (5 bytes/3MBs is ~17us send time.)
ie you rely on the hardware buffers, which are ~ 512 bytes on FT231X / CP2102N devices, to smooth.
Careful pairing of WAIT and POS detector / NEG detector hardware modes, could capture one channel down to quite narrow pulse widths (<< send time), and with granularity equal to SysCLK precision.
You NEED one. Maybe I do too... :-) What is worse, we can't blame this on erco!