Tim Moore's pcFullDuplexSerial4FC with larger (512 byte) rx buffer
Duane Degn
Posts: 10,588
Edit: See below for 512 byte buffers.
Tim Moore's pcFullDuplexSerial4FC has made my life much easier.
In case you don't know, pcFullDuplexSerial4FC uses one cog to control up to four serial ports. It's great.
After studying the code for a while, I saw how Tim reused the variable space in the hub after the cog is launched. After tracking down which variables were used by both Spin and PASM and which ones were only used within the cog, I realized there was even more space that could be reclaimed.
At the cost of three longs, I was able to double the rx buffer from 64 byte per port to 128 bytes per port.
I'm pretty sure the cog can still be stopped and restarted. I haven't tested this yet.
I'll probably modify this code again to make the rx buffer 512 byte each. This shouldn't take up much memory but the PASM code will be overwritten so if the cog is stopped, it wont be able to be reloaded. (This isn't a problem for me. I usually leave the cog running.)
I'm hoping:
1) This is useful to someone besides me.
2) Others could test it out and make sure it works correctly.
(I've tested it with 128 byte messages sent at 57600bps but I haven't tested the flow control aspects of the object.)
3) I'd prefer Tim includes my changes in object rather than having two versions of his code floating around. I'm not sure if I'll upload this to the OBEX. I'd feel kind of funny since most of this is other's work.
I'm excited this seems to work. Hopefully I can start giving back a bit for all the great stuff I've taken from these forums.
One thing I tried and failed to do, was turn this into an eight port object that uses two cogs. I thought I'd changed all the necessary pointers to a second group of buffers when launching a second cog, but it didn't work. I might try again but if any of you manage it, please let me know.
Duane
Edit(3/11/15): I have deleted the file pcFullDuplexSerial4FC128.spin
and pcFullDuplexSerial4FC512.spin
I plan to upload these programs or an improved version to my GitHub account
If there isn't a replacement on GitHub send me a message and I'll make sure to upload the replacement code.
Tim Moore's pcFullDuplexSerial4FC has made my life much easier.
In case you don't know, pcFullDuplexSerial4FC uses one cog to control up to four serial ports. It's great.
After studying the code for a while, I saw how Tim reused the variable space in the hub after the cog is launched. After tracking down which variables were used by both Spin and PASM and which ones were only used within the cog, I realized there was even more space that could be reclaimed.
At the cost of three longs, I was able to double the rx buffer from 64 byte per port to 128 bytes per port.
I'm pretty sure the cog can still be stopped and restarted. I haven't tested this yet.
I'll probably modify this code again to make the rx buffer 512 byte each. This shouldn't take up much memory but the PASM code will be overwritten so if the cog is stopped, it wont be able to be reloaded. (This isn't a problem for me. I usually leave the cog running.)
I'm hoping:
1) This is useful to someone besides me.
2) Others could test it out and make sure it works correctly.
(I've tested it with 128 byte messages sent at 57600bps but I haven't tested the flow control aspects of the object.)
3) I'd prefer Tim includes my changes in object rather than having two versions of his code floating around. I'm not sure if I'll upload this to the OBEX. I'd feel kind of funny since most of this is other's work.
I'm excited this seems to work. Hopefully I can start giving back a bit for all the great stuff I've taken from these forums.
One thing I tried and failed to do, was turn this into an eight port object that uses two cogs. I thought I'd changed all the necessary pointers to a second group of buffers when launching a second cog, but it didn't work. I might try again but if any of you manage it, please let me know.
Duane
Edit(3/11/15): I have deleted the file pcFullDuplexSerial4FC128.spin
and pcFullDuplexSerial4FC512.spin
I plan to upload these programs or an improved version to my GitHub account
If there isn't a replacement on GitHub send me a message and I'll make sure to upload the replacement code.
Comments
I do think the current name is very unfortunate.
I would never have guessed what it was had it not been for falling over it in another thread, It is a four port serial full duplex module why not name it as such. I nominate "FourPortSerial"
If you need to be able to relaunch the cog use the 128 byte buffer version.
There is still some room in the cog. I'm thinking of adding a flag for each port to indicate when a carriage return (or other character) is received.
I personlly like to have big rx buffers so I don't lose any information coming in while I'm taking care of other data collection and storage tasks.
I attached the 512 byte buffer version to the top post.
@Dr_Acula, Weren't you recently looking for a larger buffered serial object?
@Ray0665, I'm with you on thinking this could have a better name. The FC at the end stands for "Flow Control." I haven't tested to make sure the flow control still works properly with my changes. I don't think it's my place to change the name of the object. I decided to add a number to the end indicating the number of bytes the rx buffers have. (The original has 64 bytes per port.)
I hope this helps others. It's going to be very useful to me.
I'm attaching an object I used to test the vaious serial objects.
If you use it with objects having smaller buffers, you'll want to comment out some of the output commands so the buffers don't over flow. I've left comments at the end of the various test strings to indicate their sizes.
Duane
Attachment not found.
Edit: The 512 byte buffers version is 52 longs larger than Tim's original code.
The 128 byte version is 3 longs larger than the original.
Edit(3/11/15): I have deleted the archive MultiRxTx110222a - Archive [Date 2011.02.22 Time 14.32].zip
I plan to upload this program or an improved version to my GitHub account
If there isn't a replacement on GitHub send me a message and I'll make sure to upload the replacement code.
The com lines seem to work better with pull-up resistors on the rx pin (I use 10k ohm resistors).
You'll need to connect the appropriate rx and tx lines in the test object.
I don't see why the com lines should work better with pullups on the rx lines. Are you sure you aren't something to make it an open mode instead of driven? That is another great feature of Tim's driver, that it supports all variations of open and driven modes and well as true and inverted, with handshaking. The open modes require resistors.
I agree that the name of the object could be more descriptive.
The one I use was written by Pullmoll and is a modified version of Tim Moore's object that has 2 serial ports and 256 byte buffers.
How are you going to get 512 byte buffers? x4 that is 2k, which is the entire memory in a cog?
I'm using a PPDB to test the program. Using the previously attached test program I have 2k resistors between one port's tx pin and another's rx pin. I found I was receiving lots of $00s with my modified objects. I changed the serial object back to Tim's original and I still had a bunch of zeros coming in. Since Tim's code also seems to produce the zeros, I don't think my modifications were causing the problem.
I think I've just ignored zeros when I've used Tim's code in the past.
I've used a similar technique of transferring incoming messages into a different buffer. I too included some processing while transferring. I'd ignore certain characters and watch for others. I thought a better technique would be to have the serial object with a larger buffer instead of having the data pass through two buffers. I'm trying to make another modification of having the characters tested within the serial cog and to set flags when various characters are received (this isn't working yet). If I can eliminate the need of shifting incoming data, I think I can save myself a cog in many of my projects.
@Dr_Acula, I'm pretty sure the buffers are all in hub RAM. The cog running code writes each byte to the hub as it is received. (Someone please correct me if I'm wrong.) So the cog's size isn't a limitation of buffer size. The previous buffers used memory located within the cog's image giving it an appearance of being within the cog. The buffer could be completely outside the cog image. (This was my original goal, so I could use the same PASM code to launch two cogs with eight serial ports. I haven't gotten this to work(yet).)
The beginning of port zero's buffer is located 58 longs before the beginning of the cog image. There are 37 longs of the cog image I didn't want to overwrite (the hub addresses of these variables are used by the Spin code of the object.) There are also 16 longs of the cog that are off limits. Adding the five longs left unused of possible cog space equals the 58 longs that fell outside of the cog image.
Since most of the cog image is overwritten by the rx and tx buffers, the cog can not be relaunched (the PASM code is destroyed). Since there isn't the possibility of relaunching the cog, I removed the stop method. The cog could still be stopped from the parent object but it would take a reboot to relaunch the serial code into a cog.
Because of I removed the stop method, the object is only 52 longs larger than Tim's original code. I think it's pretty cool to get 448 longs of increased buffer at the cost of 52 longs (another cost is the lost ability of relaunching the cog).
I modified the earlier 512 byte rx buffer object to include end of rx flags. I personally use ASCII 13 (carriage return) to indicate an end to a transmitted line.
The attached object has a "SetFlags" method. It's called by passing four characters and the address to hold the end of rx counters. It needs to be called before the Start method.
I'm hoping to free up a cog in my serial communication programs. I normally would have a cog dedicated to monitoring the incoming characters. When a 13 was received the monitoring method would set a flag in indicate a full communication line had been received. Now this monitoring and flag setting is done by the serial driver.
This will likely be my last modification to Tim's object I'll post. (It's time for me to start using these modification to get some work done.)
I plan on making a few more changes to suit my personal needs. Such as ignoring certain characters, watching for duplicate wireless messages and possibly adding some error checking.
I doubt my specific changes would be useful to others but I am willing to share them as an example of how to customize the code.
The archive I attached has a simple test program included. This newest changes seems to work okay at 57600 bps, though I'm usually just using one port at a time.
Isn't the Propeller fun?
Duane
Attachment not found.
Edit(3/11/15): Warning, the code attached is an old version. There are better options available.
I plan to upload this program or an improved version to my GitHub account
If there isn't a replacement on GitHub send me a message and I'll make sure to upload the replacement code.
Okay, I see how the buffers are ORGed 58 longs before the start of the pasm code and how they overwrite the pasm code once the cog is started. And how the rollover is accomplished at 512 bytes by the mask of $1FF instead of the original $3F for 64 bytes.
Have you been following this other thread, about timing jitter in serial objects? I'm wondering why the timing of Tim's object--at least when transmitting only--seems to be more jitter-free when all 4 uarts are enabled than it is with 1 only enabled? That seems counterintuitive, because the unused uarts are actually patched out when not used, so the processing should, it seems, run faster and with fewer bumps. The flags are a nice touch, but they do add processing time and wrlongs.
While you're at it, the code for decl needs to be updated for a small bug that was fixed in fullDuplexSerial version 1.1 --> 1.2. This has only to do with decimal display of the value of NEGX:
{ old version if value < 0 -value ' fixed bug in dec method causing largest negative value (-2,147,483,648) to be output as -0. } x := value == NEGX 'Check for max negative if value < 0 value := ||(value+x) 'If negative, make positive; adjust for max negativeI'll make the changes you suggest to the decl methods and change the attachments I've previously uploaded. I've found some comments in the 512 buffer version that no longer apply so I'll include those changes too.
I'm not sure why Tim thought he only had 8 longs left. There are lots (maybe as many as 100) longs that are left empty in the cog. The buffers are listed within the cog image but the buffers could reside anywhere within hub RAM. A lot of other functions could be added to the cog.
The fasted baud my equipment uses is 9600 bps. PASM is so much faster than Spin, I think I can check each byte as it is received and still keep up with a 9600 bps stream. The addition of checking for an end of line charater hasn't seemed to interfer with the driver reading data at 57600bps. I figure I should be able to add all of my data checking within the serial driver to free up a cog.
I need to finish this yak shaving and get back to work.
I have been following the other thread about timing jitter. I don't think I know enough yet to contribute.
I'm still curious about the pull-up resistors. You can use Tim's code without them and not receive a bunch of zeros? Makes me wonder what I'm doing wrong.
I'm transmitting from one pin to another on the same Prop with a 2k resistor. Are you using a RS232 line, or some other hardware that would hold the line high?
This version has the larger 512 byte rx buffers, it watches for a byte indicating the end of a message similar to an earlier version I've posted. The previous version that checked for an end of message indicator could use a different indicator for each port. This new version, the same indicator is used on all four ports (so I could fit my new PASM additions without having to drastically rearrange the buffers).
The first port of this new version also checks to see if incoming data is an acknowledgment. If an acknowledgment is found, the driver sets a flag so the calling object knows the message has been acknowledged without the need of looking through the rx buffer.
I've explained some of my reasons for wanting to do this in this thread. I tried to explain what I'm doing in posts #7 and #10.
In hindsight, this probably wasn't the best use of my time. I think Dave Hein is probably correct in thinking the acknowledgments could have been handled with Spin just fine. Oh, well. But since I have the acknowledgment in the PASM section I might as well use it.
This modification is intended to be used in two Propeller boards. The object I posted uses one Propeller to test it so there are two instances of the driver object used. Even through the acknowledgment flag is set within PASM, at least one of the Propellers using the driver needs to check the rx buffer for new messages as it waits for an acknowledgment. Otherwise if both Propeller send each other a message at nearly the same time they will both get stuck waiting for an acknowledgment. In my demo program one cog checks the rx buffer while waiting for an acknowledgment, but the other cog does not.
Part of the acknowledgment system I'm using relies on a variable setCount. The setCount variable is incremented with each message sent. This byte sized number is the fifth character sent in the original message. The receiver then sends a message starting with the designated acknowledgment character (I'm using the character "R" which is hard coded in the driver to save room.) The setCount value is the second character of the acknowledgment message. The third and final character of the acknowledgment message is a carriage return (13). I use this method since I've found many of the messages are not successfully received or the acknowledgment is not successfully received. This way the Propeller receiving the message knows if it is a new message or if it is a retransmission of the previous message.
I plan to use these driver with XBees. My XBees are Series 2.5. I commented out the calls to the XBee specific methods. I left these methods in the code in case anyone else wants to use it with XBees. I'm not sure how the code would work with Series 1 XBees. If you do use it with XBees make sure and read the comments in the DAT section of the top object about serial numbers.
You should be able to test this code by following the wiring instructions in CON section of the top object.
I initially intended to use the flow control features of Tim's object but I removed the flow control part in an attempt to find a bug. The bug was not related to flow control but I have not tried to use again.
This test code has a lot of rough edges. Sorry, but I didn't want to take the time needed to clean it up. I need to put this driver to work collecting data.
Edit: I had the wrong version uploaded. This should be the correct version.
XBeeComTest110304b - Archive [Date 2011.03.04 Time 14.52].zip
Edit(3/11/15): Warning, the code attached is an old version. There are better options available.
I plan to upload this program or an improved version to my GitHub account
If there isn't a replacement on GitHub send me a message and I'll make sure to upload the replacement code.
There's nothing wrong with putting the acknowledgments in PASM. Someone might want to use your code in the future to do high speed transfers, and acknowledgment in Spin may not work at that speed. I have some YMODEM code in Spin that just barely handles 115,200 baud by using a large receive buffer. I could run it faster with a smaller receive buffer if I moved some of the code to PASM.
Dave
I agree there isn't anything wrong with putting the acknowledgments in PASM. But I don't think they were necessary in this case. I think my new PASM programming skills were going to my head and wanted to try to add things I'd normally do in Spin to the PASM section.
To really get the acknowledgments to work quickly the acknowledgments shouldn't just be watched for in PASM but also sent from PASM. This is a can of worms I'm not ready to open.
I'm excited to have access to the power and speed of PASM. It's made programming the Propeller even more fun.
Duane
Has anyone figured a way to make it operate reliably at 115,200 baud?
I tried it but soon discovered unacceptable timing problems that caused me to lose data.
When I slowed it down to 57,600, it worked fine.
I've followed the other thread on timing accuracy. Is this just asking too much for a 4-port object in one cog?
Jim
You're probably right, that kind of speed is too much to ask for from a four port object in one cog. Well, not too much to ask for, just too much receive.
Some of the modifications I've posted slow down the object. But the objects in the top post shouldn't make it run any slower. You can have a 128 byte rx buffer for free and a 512 rx buffer if you don't need to stop and then relaunch the cog after it is first started (it also increases the code by 52 longs).
I know there is a two port driver out there. It might be able to operate at 115,200 baud.
This time I increased one of the TX buffers.
I use an Emic text to speech module with my data logging Propellers.
I had the problem of the Propeller having to wait for the Emic to finish saying things before continuing its main loop. One solution was to have a "FeedEmic" method in a separate cog. Income data would be sent to holding buffer and the "FeedEmic" method would watch the Busy pin and send a new line of text when the Emic was ready.
But since I recently used the flow control features of Tim's object (which I apparently didn't ruin) to send data to a XBee using its CTS line, I thought I could treat the Emic's Busy pin as a CTS flow control. It worked. This saves me yet another cog.
The problem I had was the Emic processes its data much slower than a XBee and so it would hold up the program once the Emic's 128 byte buffer and the 16 byte TX buffer were full. I don't think the Emic usually uses the full 128 bytes. That's just the longest line of text it will accept. I'm pretty sure the Busy pin goes high when it receives a end of line character.
So the solution was to increase the TX buffer. My first attempts weren't successful as I point out here. Once I fixed my dumb mistake (which I make more frequently than smart mistakes), the increased TX buffer worked great. The Emic's Busy pin acts as a CTS line and all seems well.
I think it shouldn't be too hard to for someone to look over the changes I've made and customize Tim's object for their own needs. I think the buffer doesn't even need to be a power of two. One might need to multiply instead of shift bits to send or read data to/from the correct buffers but I don't think these changes would be very difficult.
The rx and tx buffers don't need to be within the cog image. Tim and I have put them here just to save on hub RAM. This most recent change uses a tx buffer completely outside the cog image (since that space was all ready used up).
Sorry, but I didn't take time to clean up this code. There are all sorts of extra methods I used to debug the program.
I included the Emic methods in with the serial driver. This way the Emic is one of the four ports (hard coded as port 3 for now). The Reset pin number is a parameter in the init method.
The str method adds "say=" to the beginning of any string sent to port 3. I'm not sure if I like this approach but be warned it's there for now.
I think I have freed myself of needing (as many) extra text buffers and I can make much better use of the cogs in the Propellers needing serial lines.
One more warning. This version uses an incrementing flag that counts end of line characters received. I use these counter to see if the Prop has received new messages. Although come to think of it, in this particular case the only receiving being done by this test program would be from the terminal program.
EmicComTest110316a - Archive [Date 2011.03.16 Time 09.05].zip
Edit(3/11/15): Warning, the code attached is an old version. There are better options available.
I plan to upload this program or an improved version to my GitHub account
If there isn't a replacement on GitHub send me a message and I'll make sure to upload the replacement code.
Have you any luck with multiple ports at 115.2k? I don't need to have four ports per cog -- 2 is fine, and I'll just use 4 cogs up if that makes 115.2k more reliable. Once I have my wall wart I'll be ready to start testing stuff. Oh wait, now I also need a MAX232 to interface with my PC... drat. Are you guys doing your testing by looping TX to RX instead?
You could try just adding two or three ports before calling the Start method and test how many you can add and at what speeds.
There's a test program in most of the archives I've uploaded.
Duane
I don't need wired serial routing right now, but I think I might use some of your ideas on a wireless router I've been thinking about.
Does this mean some of my modifications will be in space? If so, that is so cool. If not, it's still cool just not as cool as some of my code ending up in space.
Duane
As to whether the "big" rx buffer or not will be used, I honestly don't know, I wrote two versions and they'll use whichever is deemed more stable.
I also looked at the issue of CTS being the wrong polarity, which is another issue that has come up a couple of times. That is, the state of CTS that allows flow to take place should be the same state as the start bit on the TX line.
The following highlighted in red reverses the CTS patch, while the blue highlights making RTS an output. This is in the initialization code for port zero (one added long). The same adjustment needs to be made to the init code for all 4 ports.
[SIZE=1][FONT=courier new] or ctsmask,#0 wz 'cts pin? z not set if in use if_nz test rxtx_mode,#INVERTCTS wc 'c set if inverted [COLOR=#ff0000] if_nz_and_c or ctsi0,doif_z_or_nc 'if_nc jmp TTA, reversed the sense of these two patches, and_c <_> and_nc if_nz_and_nc or ctsi0,doif_z_or_c 'if_c jmp [/COLOR] if_z mov txcts0,transmit 'copy the jmpret over the cts test if_z movs ctsi0,#txcts0 'patch the jmps to transmit to txcts0 if_z add txcode,#1 'change co-routine entry to skip first jmpret 'patch rx routine depending on whether rts is used 'and if it is inverted or rtsmask,#0 wz 'rts pin, z not set if in use if_nz test rxtx_mode,#INVERTRTS wc [COLOR=#020FC0] if_nz or dira,rtsmask ' TTA rts needs to be an output (will start in whatever output state) [/COLOR] if_nz_and_nc or rts0,domuxnc 'patch muxc to muxnc (TTA, this is okay as is) if_z mov norts0,rec0i 'patch to a jmp #receive if_z movs start0,#receive 'skip all rts processing if not used [/FONT][/SIZE]I'll try to make time to update my modified versions.
Here's the thread were Ron Czapala talks about this issue.
I didn't know about the polarity of CTS lines. I think I've only used flow control with XBees. I think I tried to use it with an Emic but it didn't work correctly (probably because of this very problem).
Does the CTS line on one device connect with the RTS the other? One is an output and one is an input, right? If I remember correctly, CTS is the input.
Sorry for such basic questions my efforts to find this information online have failed so far.
"Does the CTS line on one device connect with the RTS the other?". That is a can of worms! It depends on how the pins are labeled, and it is traditionally done from the standpoint of the data terminal equipment (DTE) as it talks to data communications equipment (modem, DCE). So RTS was an output from the DTE and an input to the DCE. Vice versa for CTS. So, RTS to RTS and CTS to CTS. But then if you connect two DTEs (computers) together, you would need a null modem to cross-connect RTS to CTS. Whole books have been written on the subject. What it boils down to is that you have to take a step back and read the data sheet or probe the pins to find out which pins are outputs and which are the associated inputs, no matter what the labels may say.
As to polarity, usually non-inverted TX and RX go with non-inverted RTS and CTS, but that is not a hard and fast rule either.
Here is a revision with the 128 byte buffers patched to enable RTS and to correct the polarity of CTS.
Also I attach a test program that chains a message from port 1 -> port 2 -> port 3 and then to debug port 0. The RTS and CTS are set up for testing on port 2.
-- With a 'scope look for a high pulse on RTS when the port 2 receive buffer fills past the byte-count threshold.
-- Connect CTS to Vdd to stop flow, and to Vss to allow flow.
-- After every 10 iterations, the program restarts the ports, just to verify that it works. Restart is important to my apps.
By the way, I understand now what you mean about needing pullups the inputs. It happens when inputs are floating. On most(?) Prop boards, floating inputs tend toward a low level, and fullDuplexSerial in "true" mode treats a continuous low as a succession of null input characters. Maybe it should treat them as framing errors and flag them,not put them in the buffer, or flag a BREAK, but that is another issue.
This is great. Thank you.
I've been thinking a lot about this project ever since your PM concerning it. Someday soon I hope to spend some time polishing up Tim's driver.
I'd like to think I've learned a lot since I first attempted these modification. I think I've got the circular buffer thing figured out now.
I've made a habit of always using pull-ups on my data lines because of the null character problem. I suppose they could be screened out by the low level driver (I'm pretty sure I could do this) but aren't there times you'd want to transmit a null character?
Thanks again for your fixes. This project is making its way back up to a front burner project.
One option I'd like to add (which I unsuccessfully attempted a year ago) is to have an option of adding up to eight ports by using more than one cog. At present you have to make a change in the program code (not just the title) in order to use a second instance of the object.
I'm curious why restarting is such an important feature? Is so you can reduce power by turning off cogs?
The reason I want to keep it restartable is that one of our apps involves a modem that may have an incorrect baud rate, and being able to restart makes it easy to scan through the possible rates until it responds and can be reprogrammed the way we want it. It also has to do with turning off cogs to save power, and restarting them at a drop of the proverbial hat.
There is a difference between a null character and one that is missing its stop bit (framing error). One should be able to receive nulls, certainly. BREAK is a sustained input at the level of the start bit (longer than a byte, longer than a framing error, even out to seconds), and that is used for things like waking up peripherals or sending a reset signals, such as is done at the commencement of Propeller or BASIC Stamp programming.