Serial Com with buffers

Rsadeika · 2009-08-22 15:33

Is anybody working on a program like FullDuplexSerial that would have TX/RX buffers? I am thinking that the buffer sizes could be assigned at initialization of the Com program, and would allow for different buffer sizes for the different Com programs that you would start.

I would like to have a scenario where, lets say, you have, in the RX buffer, the string - "Turn on LED 16". I would like to be able to take that string, that is in the RX buffer, evaluate it, and do a corresponding action. I will be running at least three com programs, which will have their own buffer size assignments. Any ideas?

Thanks

Ray

JonnyMac · 2009-08-22 16:06

FullDuplexSerial has buffers -- albeit small. That said, with a few changes to the code you can make them as big as you like (must use powers of 2).

Timmoore · 2009-08-22 17:15

All the fullduplexseral versions have buffers, the default is 16chars but its easy to change to a power of 2. There are versions with larger buffers available as well. The 4 port serial version have buffer sizes of 64 chara per port. That ones not easy to change - there is no space to make it bigger.

Rsadeika · 2009-08-22 18:28

OK, after looking at the listing for FullDuplexSerial.spin, I found this:

VAR
byte rx_buffer[noparse][[/noparse]16]
byte tx_buffer[noparse][[/noparse]16]
This must be the small buffer everyone is referring too. I also found the methods where these are being used. Do I surmise that at all times these arrays are filled with bytes, up to sixteen of them? Of course there would have to be a stream of bytes coming in.

There is also a method, PUB rx : rxbyte. I realize that this pulls one byte at a time, so where would it be best to handle a string of bytes, within the FullDuplexSerial method, or within the object that is calling it? In the example - "Turn on LED 16", I have to deal with fourteen bytes, can I just access the rx_buffer[noparse]/noparse, and get it there? Or is their a better way of doing it?

JonnyMac · 2009-08-22 18:55

The RX buffer is filled by the "background" UART code; you use the .rx method to pull one byte at a time from the buffer.

What you're wanting to do seems very simple on the surface but is complex -- you're trying to get the Propeller to do natural language processing. What about case? What about the LED on pin 15? What if the user enters "Turn LED 16 on" (order of words is different). These things are tough. You can simplify them, though, by creating a shortcut command syntax. For example:

L 16 1

could be used to turn the LED ("L" command) on P16 on (1). This simplifies your parsing of the incoming stream. After you see the letter command you can jump to a routine that handles it. In the case of the LED command it is expecting two decimal values: the pin number and the state (1 or 0).

Rsadeika · 2009-08-22 19:15

Yes, JonnyMac, I want to see if I can do some "natural language processing" on a very very very small scale. I would like to see what it would take to process "Turn LED 16 on". First I have to figure out how to deal with more than one byte at a time, that is delivered via FullDuplexSerial object.

Timmoore · 2009-08-22 19:17

The other thing to work out early is how the 2 sides synchronous where they are in a command.
What is the start of a command, the end?
What happens is the prop is started after the otherside or the other way round?

A common approach is the start/end to be the newline. The app initializes by calling rx and discards until it receives a newline, then the main loop calls rx and puts the char into an app buffer until it gets a newline. Then it processes the command in the app buffer, then it loops back to calling rx and putting the char in the app buffer.
With this approach you miss the 1st line but you dont try to process partial commands. If the otherside is also a program it sends "I", newline (init command) until the prop sends a "O", "K", newline response then the 2 sides are in sync.
This way the serial port rx buffer is not the buffer the app is using, its used by the serial port to buffer when the app is busy and chars are arriving. The app buffer is used by the app to process commands.

I have attached an example - GPS processing. GetNMEAline is the main processing routine, its calling rxcheck, it has 2 states - waiting for start of line - discarding until it sees a $ which is GPS start. Wait for end of lilne - receive char put in buffer, if end of line call ProcesNMEALine and then go to wait for startof line, otherwise stay waiting for end of line.
ProcessNMEALine looks at the gps buffer and works out the commands to process.

rxcheck is used rather than rx so it doesn't wait, if it doesn't receive a char it can go and do something else

Post Edited (Timmoore) : 8/22/2009 7:24:34 PM GMT

JonnyMac · 2009-08-22 19:21

You're going to have to pull a one string at a time from the buffer -- and do that one character at a time. If you see that you have characters in the buffer then you start pulling them, moving them to another place (an array) until you hit a whitespace character. Now you have a string you can work with.

This is going to be tough, but if you really want to give it a go then you should have a look at Mike Green's Femto BASIC interpreter for ideas on parsing strings and working with them.

Nick Mueller · 2009-08-22 19:27

End the phrases with a CRLF!
This way, you get char by char from the serial until you see a CR/LF. That's the signal to stop reading and start processing the string. After that, start reading again ... and so on and so on.

To change the buffer size, you also have to modify the PASM. It's all not commented. Neither in SPIN, nor in PASM. It's not enough just to change the buffer size. If you want to handle and RvBuffer and an TxBuffer of different sizes, you have to modify the PASM even more. See comments.

Nick

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Never use force, just go for a bigger hammer!

The DIY Digital-Readout for mills, lathes etc.:
YADRO

StefanL38 · 2009-08-22 19:34

the Extended_FDSerial-object has additional methods that can receive strings

take a look at that

best regards

Stefan

Rsadeika · 2009-08-22 19:53

I knew that I would have to use some kind of scheme for parsing strings. I guess it looks like I use FDS to grab the string, and then parse it within the main object. So, FDS stays as it is for now, and I will get a feel for parsing strings; this should be fun.

Yes, the phrases will be terminated with a CR/LF. I am also developing a terminal program, written in freeBASIC, which will handle the other end of the conversation. So, my first objective is to have my terminal program on my PC converse with a Propeller demo board using BlueTooth as the conduit. Hence the phrase "Turn on LED16".

So far what I have is, on my terminal program, I press the letter 'a' to turn on/off LED 16, and 's' to turn on/off LED 17, I will add a CR/LF to this. And the Demo board does the actual work of completing the instruction. Maybe I will add in a response from the demo board side, "On a coffee break, will be back in 15", just to lighten things up a bit.

JonnyMac · 2009-08-22 22:40

For fun I wrote the attached program to control LEDs on the demo board. It uses single letter input but does allow you to enter timing (in milliseconds) for certain commands.

Kye · 2009-08-22 23:39

Mmm, why does no one ever use my serial engine contribution where I·took on·these type of request in detail.

http://obex.parallax.com/objects/397/

Everything you need. Read·the documentation·for the receiving functions and you will understand.·Much smaller than fullduplex serial also.

...

Use the attached file for string processing.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Post Edited (Kye) : 8/23/2009 3:41:38 AM GMT

JonnyMac · 2009-08-23 00:23

It's really tough to keep up with what's in ObEx and not all of it is worth using. Perhaps you could show Ray how to use your library is by writing a little demo program that solves his specific problem, e.g., parsing a string like "Turn on LED 16" -- and making it happen on a demo board.

Kye · 2009-08-23 03:55

Simple enough. Extend this program for your needs.

Its does exactly what is asked.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Rsadeika · 2009-08-23 15:23

@Kye, Thanks for taking the time for putting together some code. I had looked at your serialengine object code a while back, and the problem, for me anyway, is, that you have RX_pin/TX_pin assignments hard coded. Now, you are probably thinking, how lazy is this person, just cannot make some changes to the code to suit their own needs! I am not an asm person, so I know that if I start to change things in the asm code, for sure, I will mess things up. Although, I must say, that if I were to pursue asm, I would use your serialengine object as a starting point.

At this point I am trying to show Proof Of Concept (POC), and I need to go at it in the most proficient manner as possible. Later, I can think about making algorithms more efficient, by converting them to asm. I am also involved with my PC terminal program, which is eating up a lot of time, but is necessary because it will be designed to work with the Propeller, specifically my Propeller object code.

At this point I would like to mention the other serial program, the one that uses one cog for multiple UART instances, may be a good solution, but ... It lacks some clear documentation, and some examples, as to how it should be used. That alone stopped me in my tracks. So, I have looked at most of the objects that are available, and each one has their own little niche that it fills, but does not fill my niche. I guess my POC will be a little longer to achieve, I will probably have to come up with my own serial object.

Thanks

Ray

Rsadeika · 2009-08-23 20:53

After looking at UARTEngine.spin object, I just simply changed the RX_Pin/TX_Pin values to the new pin settings that I am using, the program seemed to work. Very good job Kye. The only problem is that when I change the RX/TX pins I have to remember to go and change the code. Since I mentioned that I will have at least two more UARTS running, this could get a little hazardous if I start changing the RX/TX pins in a hurry. Plus, I will have to have three different named UARTEngine.spin objects. But, I guess I will get a savings in cogs used; as I understand it FDS uses a a cog every time you create an instance of it. If you have fds[noparse][[/noparse]0], fds, and fds, that would be using three cogs plus one cog that the FDS object itself uses.

Here I thought I would have to get into the asm code, to my delight, the problem was solved in quick order. Now, I will look at your stringEngine object to see what you got going there.

Thanks

Ray

jazzed · 2009-08-23 21:24

It seems like a generic solution that processes received strings based on a subset of regular expressions would be useful. That would mean basically having a FullDuplexSerial type of object with a big receive buffer and a "rules" queue that could be populated by the users. A fair starting point for understanding Reg-Ex is Wikipedia en.wikipedia.org/wiki/Regular_expression . Of course this could be a little old-school and cryptic for some, but is nevertheless useful.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
--Steve

Propeller Tools

Kye · 2009-08-24 00:39

@jazzed - my driver was made to fix all the problems you noted above.

@Rsadeika - my driver does not save you a cog. It still uses one every time you call uartEngine to start the driver. You will need X# versions of the driver each all with their own pins configured to different pins.

In that sense my driver is no better than FDS, however you will find mine uses less code space, has much larger buffers, and is much faster if you peform a raw speed test.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Rsadeika · 2009-08-24 13:49

To broaden my knowledge base, I am looking at the Simple_Serial object. In the info segment it states that it has been tested to 19.2 kbaud; so, naturally, I tried it with 57600 kbaud, which did not work, or at least I did not have anything displayed on terminal screen. I am wondering, what would it take to get the program to run up to 57600 kbaud and beyond? Looking at the code, I am not recognizing, what it would take to get it to work with 57600 kbaud. If it has something to do with the stop bits, then I am in way over my head.

Thanks

Ray

StefanL38 · 2009-08-24 14:26

No the simple reason is the limited speed of interpreted SPIN-code
interpreting a SPIN-command needs time to load the SPIN-token, fetch parameters, branch to the underlying assembler-routine, and execute the assembler-routines
and all of this takes time.

As a maximum speed-test how fast SPIn is executed you can code a small loop simply toggling an IO-pin low-high-low.... and measure the frequency

  repeat
     !outa[noparse][[/noparse]10]

or if you would like to measure execution-time of other commands

  repeat
     outa[noparse][[/noparse]10] := 1
     'your commands ......
     outa[noparse][[/noparse]10] := 0

at 19200 baud the high-time of a bit is long enough to be reliably detected it as high when coding in spin
above this baudrate you will miss some of the bits

best regards

Stefan

jazzed · 2009-08-24 17:11

Kye, I did not note problems. I noted a potential solution to receive token detection, etc... specified by user; clearly, this is not in your driver, but it could be.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
--Steve

Propeller Tools

Rsadeika · 2009-08-24 17:52

I think that I am going to go with the Kye UARTengine, it seems to be about the easiest, of the bunch, to work with, and to build on if necessary. The stringEngine also gives me a good starting point, but I do see some changes that will be implemented. The UARTEngine is a little bit cumbersome, due to the fact that I will need at least three of them going at the same time, but I will work around that. In terms of cogs, it will not be a problem at the moment, but I think I will have to add another prop, to beef up the amount of accessible cogs. Now I wish the Propeller had sixteen cogs to work with. Since there are some memory boards that are starting to appear, I do not find that the 32KB on the Propeller, to be that much of a limitation, for my project I am finding the availability of cogs to be a limitation.

Kye · 2009-08-24 23:31

@jazzed What?

I have the receiveCheck method. It lets you peek at the last entered character into the receiver buffer. Good for looking for line feeds or carriage returns. You aren't sopposed to do string processing on stuff within the serial driver's com buffers.

That would be very bad. Instead you transfer the string to the string engine's buffers. There, you can do all the string manipulation you want.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

jazzed · 2009-08-25 00:01

Generally as you may have learned in school, handling things at an "interrupt level" is bad especially in a non-preemptive O/S. However, when called upon, you handle things in your driver when it is fast enough to handle it and the user is not [noparse]:)[/noparse] which is the case between pasm and spin. The more you can do for the user in pasm the better. This is similar to many other cooperative implementations like Java/native functions, C/ASM, Python/C, etc... Consider the speed disparity imposed by spin over what can be provided. Your receiveCheck() is at least standard fare as provided by any libc gets() or system.* readLine() like method, which I'm sure is appreciated. Still, you're doing fine.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
--Steve

Propeller Tools

Kye · 2009-08-25 02:40

Yeah the method is pretty much made to be used in a tight repeat until loop doing nothing other than checking for a new string. It would waste more code space however to give that function to the pasm. The spin runs it wuite well in a tight loop.

Using the propeller architecture however, your program should use one cog to only poll that buffer and nothing else waiting for input. Another cog should be used to run other parts of the program.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Serial Com with buffers

Comments