PropTCP Beta - Now fully MIT Licensed (w/ AJAX Enabled HTTP Server Example!)
Harrison.
Posts: 484
I would like to announce that PropTCP is now completely MIT licensed. I have decided that it's in the best interest of the Propeller community that all code be released under the MIT license. GPL compliance was impossible to enforce anyways, especially for small projects like PropTCP (violators probably wouldn't have cared about licensing anyways).
PropTCP now ships with a simple HTTP server example which only requires an ENC28J60 interface (configured for the standard YBOX2 hardware). The server supports 3 'simultaneous' connections, although it can only actually service one request at a time (the other requests are queued). The HTTP server serves up an AJAX enabled page which constantly displays a random number from Propeller. It also supports downloading of the Propeller HUB ram for speed test purposes. I am currently working on porting an older HTTP server example which serves files off of a SD card (it also has a 'full' featured file manager for file uploads and downloads).
I have attached my latest version. It is still in beta, but I am actively testing it on multiple Propeller boards to ensure that it is stable before I post it to the OBEX. I would appreciate any feedback anyone has.
There have been many updates to PropTCP over the past year. Below is a summary of changes:
Here's a short listing of examples to-be-released:
Post Edited (Harrison.) : 12/8/2009 10:07:04 AM GMT
PropTCP now ships with a simple HTTP server example which only requires an ENC28J60 interface (configured for the standard YBOX2 hardware). The server supports 3 'simultaneous' connections, although it can only actually service one request at a time (the other requests are queued). The HTTP server serves up an AJAX enabled page which constantly displays a random number from Propeller. It also supports downloading of the Propeller HUB ram for speed test purposes. I am currently working on porting an older HTTP server example which serves files off of a SD card (it also has a 'full' featured file manager for file uploads and downloads).
I have attached my latest version. It is still in beta, but I am actively testing it on multiple Propeller boards to ensure that it is stable before I post it to the OBEX. I would appreciate any feedback anyone has.
There have been many updates to PropTCP over the past year. Below is a summary of changes:
+ [b]Flexible Buffers[/b] - Buffer arrays are now supplied by the user application, which allows for different buffer sizes per socket. Buffer sizes must still be powers of 2.
+ [b]TCP Retransmit[/b] - The sockets layer now performs retransmission whenever a packet is lost.
This results in very reliable communications. Unfortunately this also decreases thruput on high latency links.
+ [b]RST Timeout[/b] - All socket states now have a timeout to prevent lockup issues.
+ [b]Thread Safety[/b] - Public methods are now all thread safe. This means you can call them from different cogs without the previous lockup and corruption issues.
+ [b]Handle Validation[/b] - Socket handles are now validated using socket state information.
+ [b]Improved Error Codes[/b] - Better error codes are now returned.
+ [b]Speed Improvements[/b] - The stack can now transmit data at up to 70KBytes/sec. This is dependent on link latency and buffer sizes.
+ [b]Smaller Code[/b] - I refactored a bunch of code to improve efficiency. This in turn reduced compiled code size, and increased speed by roughly 15%.
+ [b]MIT License[/b] - The code is now MIT licensed for OBEX compatibility!
Here's a short listing of examples to-be-released:
+ [b]HTTP Bootloader[/b] - Program your Propeller over the internet (similar to Darco's YBOX2 bootloader, but hardware independent) + [b]HTTP Webserver[/b] - A webserver with a SD card filesystem, web based file manager, and user loadable SPIN binary applications (should be better than the Parallax PINK...hopefully) + [b]DNS Resolution[/b] - Resolve domain names (using TCP-based DNS queries w/ OpenDNS)
Post Edited (Harrison.) : 12/8/2009 10:07:04 AM GMT
284 x 172 - 11K
Comments
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)
· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm
Thanks for this, Harrison.
It was this time, last year, that Ethernet and the AVRs pointed me to the Ybox2 and hence the Prop. There will be so many projects for the Xmas break
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Style and grace : Nil point
I just took a quick glance at the driver, and I have a couple of questions:
1) You require a pin for INT, but it does not seem to be used anywhere... must it be hooked up?
2) You provide a 25MHz clock to ENCJ, what changes, if any, are needed if a 25MHz crystal is provided for the ENCJ? (other than removing the prop generating the 25MHz clock)
I am trying to see if I can use your stack with just four pins connected: /CS, CLK, DO, DI
Best Regards,
Bill
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheusdual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus and Proteus for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
I have used some Lantronix ethernet to serial devices in the past and certainly it has been possible for people on the other side of the world to log in from a text based terminal program. But they only come up for a good price intermittently and the price varies a lot ($5 to $200 depending on the day). Something Prop based would be much better.
I've used DynDNS to link a fixed address to my changing home address. Originally I had it working as a small progam running on the background on a PC but more recently I had a browse through my router settings and DynDNS was in the router, so I just programmed the address into my router and I can get a 'business' grade permanent address for home prices. Very handy if you want to put a board on the net for home automation etc and not have it change address every day.
Harrison, can you expand a bit more on what you are doing, as I think it might be brilliant.
Can I ping your board?
Can I log into your board, ie I send a few nominal bytes and it sends me back something like 'hello, you are now logged in'
If you can do that, you could replace the Lantronix ethernet to serial device. Or even better, incoroporate it into the myriad of single board prop computers.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Wouldn't that be great! NetDracBlade.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
The INT (interrupt) pin isn't being used right now. Just pass in -1 (or any other number) for now. I will likely remove that argument before I post the stack to the OBEX.
Pass in -1 for the xtalout pin number if you don't need the 25MHz clock. No other hardware or software changes need to be made.
Yeap, those are the only pins that need to be connected if you don't need the 25MHz Prop generated clock. Most of my boards only use those four standard SPI pins.
The stack doesn't currently support icmp ping, but that isn't difficult to add. I could probably whip up a quick demo running http and telnet. In fact, one of the earliest tests I did back in 2007 was to run Donglebasic (serial version of Femtobasic) over TCP w/ telnet.
Unfortunately I don't have any DHCP code yet. I can't figure out a good way of integrating it without using massive amounts of code space. If you really need DHCP then you might want to take a look at Darco's DHCP code for the YBOX2.
I've glanced over the rest of the files - very nice, clear to understand code.
A couple of possible memory saving suggestion:
- reuse the area used by the ENCJ initialization for buffer space
- reuse the area for the cog pasm code for more buffer space
There is also an object overlay loader somewhere in this forum... it would make it possible to dynamically load a dhcp object to obtain an IP address, then discard it. The same mechanism could be used to start the SPI cog, and to initialize the ENC28J60.
I really like the simplicity and the low cost of the ENC28J60 versus the WIZ5100 - and the ENC28J60 is readily available as a DIP part, a big plus in my opinion.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheusdual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus and Proteus for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
Post Edited (Bill Henning) : 12/9/2009 4:43:02 AM GMT
I was also working on a simple http server:
http://forums.parallax.com/forums/default.aspx?f=25&m=406015
The structure of my code isn't very nice and uses a lot of memory due to the huge bufferspace, but it's pretty fast:
90 kBytes/s @ 80 MHz Propeller clock frequency
106 kBytes/s @ 100 MHz
give 12.5MHz with pllx8.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Style and grace : Nil point
The dec function in the api_telnet_serial isn't working correctly, although the code of this function hasn't changed compared to previous versions of PropTCP.
I found out by chance, while I tried to modify my webserver to work with the newest version.
I started working on a stack of my own mainly because of the license problem. I will probably finish it because with your changes our functionality has diverged pretty far too. I am using the Spin object model to do code memory optimization that would probably save about 4K of Hub RAM if I just copied your logic. (If you define an array of Socket objects, you can get rid of *all* the dereferencing e.g. socketdata[noparse][[/noparse]pkt][noparse][[/noparse]position_in_pkt] stuff. That saves an AMAZING amount of code Hub RAM.)
I also want to optimize mine to work right with lots of fast, short connections, because that's how I have to deal with some embedded controllers I use that I can't reprogram if I want decent data bandwidth. That's how I discovered the timeout problem, something that you might notice once every day or two in a normal application was showing up every five minutes for me because my app requires opening a connection, doing your stuff, closing it, and then repeating several times a second. With the default timeouts in the usual TCP stacks and no option for UDP in the pre-programmed other device, it's the only way to get throughput. But I'd be the first to admit it's a specialty app and what I'm going to do to make it work might not be anybody else's cup of tea.
Could you post or send me your code? My demo webserver included in the first post uses .dec() to send the random number. You could be forgetting to flush the transmit buffer before closing the connection.
That's a great idea! When I first wrote the stack (Dec 2006) the Propeller hadn't been tested at speeds above 80MHz. Seems like everyone is running the Propeller at 100MHz these days (in fact, I ordered ten 6.25MHz xtals last night).
Very Interesting. The beta release posted in the first post switched from using massive pointer dereferencing to using direct arrays. I moved the packet array into the socket layer, and switched to using separate arrays for all the socket state variables. This saved a couple hundred longs (while increasing speed a bunch), but I don't think it saved as much as 1K longs. I also no longer have to do complicated pointer math to get to the socket state variables.
I ran into those connection issues about 8 months ago when I was working on a project that used a bunch of small short term connections. It became more apparent over high latency / lossy networks (like wireless). I had to add a bunch of timeouts and retransmit code to make communication reliable. The current beta should be very resilient to connection problems. I use it in a couple of mutli-cog projects, with long term and short term connections and haven't seen any issues yet.
Post Edited (Harrison.) : 12/11/2009 3:00:54 AM GMT
You might want to look into this trick since it saves even more speed and RAM; the principle is that you create a socket object with its own data and code, and in the obj block declare an array of them:
As elsewhere, the different instances of socket.spin share DAT but have their own VAR areas.· This means that socket.spin code doesn't need to use arrays at all -- the socket index is implicit in the function call...
This way, within the tcpmsg routine you don't need to use array references at all, just direct memory variables.· I also added functions in the nic object to read and write common data structures to the packet buffer and SRAM without dereferencing to [noparse][[/noparse]pkt] every time.
var byte state long srcip long seqnum long acknum '... etc. obj nic := "driver_enc28j60" pub tcpmsg case state sck_closed: return sck_listen: if (hdrflags & tcp_syn) srcip := nic.readpktlong(ip_srcaddr) srcport := nic.readpktword(ip_srcport) '...and so onOf course you need to provide functions for sck to report its state and such to the parent object, and the nic object must be available to both the parent and socket (which is no problem since it uses DAT storage exclusively).· In code blocks where I was just duplicating your functionality this was consistently shrinking it to about 60% of its original size.
I'll definitely have to try that. I've always thought that function calls were much more expensive since they have to push stuff onto the stack. I always assumed that using arrays directly were much faster since it probably doesn't need to do extra stack operations. I guess my reasoning was that it's better to have faster thruput than to have smaller code. Maybe my real issue is I need to compile with BST and look at the bytecode generated.
EDIT: Ah, I think I see what you are talking about now. You moved a lot of the state machine code into the Socket object, which allows direct variable accesses without expensive function calls / array dereferencing inside the state machine code. This sounds much better than the current method I'm using.
Post Edited (Harrison.) : 12/11/2009 7:40:44 PM GMT
I built a Ybox2, and a board of my design - the idea was that I can use the Ybox2 (known good design) to help debug my board.
Harrison or localroger - would you guys have any tcp stack debugging code kicking around that might help me?
Here's where I am at:
I downloaded the latest PropTCP beta from the first post in this thread, and made the following changes:
I am trying to get the web server demo running.
On a Ybox 2, I start with:
{ybox2} sock.start(1, 2, 3, 4, -1, -1, @mac_addr, @ip_addr)Maybe I did a bad job of soldering, because I don't even get a link light on my switch!
On my board, I start with:
{other} sock.start(23, 20, 21, 22, -1, -1, @mac_addr, @ip_addr)Here at least I get a link light!
I'm now going back to debugging...
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheusdual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus and Proteus for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
I forgot to restore the 7 when it did not work on my board and I added back in the alternate start line for Ybox2.
(WOW that was a fast response!)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheusdual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus and Proteus for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
Now I can compare it to my setup and find out where the problem is...
Thanks!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheusdual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus and Proteus for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
I tried to work on my web server again, but I still can't find the bug.
It works fine, if the data that is sent back to my computer consists only of a few bytes:
HTTP/1.0 200 OK
<html>Liste der Dateien auf dem Webserver:
DRWHO11.AVI
INDEX.HTM
INTHEEND.MP3
LICENCE.TXT
PRAKTI.JPG
PROBE.TXT
SCHUH1.M4A
SCHUH4.M4A
SCHUH13.M4A
SCHUH16.M4A
T-HOME.MP3
36GRAD.MP3
BOOTEX.LOG
CONFIG.HTM
DAVINCI.M4A
SECURITY.HTM
PIRATE.MP3
Dateien insgesamt: 17</html>
As soon as bigger files are being transmitted, this is what happens (first line is the reply of Fiddler2):
HTTP/1.0 200 This buggy server did not return headers
HTTP/1.0 200 OK
Server: PropServer
dy bgcolor="000000">
<head>
<font color="65FF85" size="10">
Testseite
</font>
</he<html>
<title>Testseite</title>
<body bgcolor="000000">
<head>
<font color="65FF85" size="10">
Testseite
</font>
</head>
<font color="65FF65">
Das ist eine Testdatei mit der ich
I know that the code of the main program looks a bit ugly, but I'm going to seperate it to several procedures soon.
Thanks for your help!
I've attached the modified version that should work very well. I had to make a few changes to make it work. The changes were:
- Added new circular buffer arrays for the .listen() call. These have to be reserved for the tcp stack, and cannot be touched by your application.
- Disabled sending of Content-Length header (it was sending 0 length because size was not set). This confused my browser (Firefox 3).
- Optimized the sd -> tcp stack transfer repeat loop. You had some unnecessary code to clean the array.
Other than that, the code seems to work very well. I can download files from an old 32MB SD card at about 125Kbytes/sec with a 100MHz Propeller clock. I tested MP3 streaming with VLC, and it works very well (even supports stopping and pausing correctly!). I guess the stack's new TCP flow control stuff works great for streaming, especially since VLC only requests a bit of data at a time.
Sorry for not making the usage of these new parameters clearer. It's my fault that you had to wrestle with this problem.
You have achieved this in a few hours, what I couldn't in several weeks
Oops, sorry... It seems I've deleted something I commented out before due to debugging. I'm going to correct that immediately.
It should work again, if you insert this line
after the "else":
if sdfat.popen(@fname,"r") <> 0 ' Try to open the requested file web.str(@http404) ' if it is not found on the sd card send back the 404 error message else size:=sdfat.get_filesize web.str(@http200) ' file has been found and will be transmitted, web.str(string("Server: PropServer")) ' but first we are going to send the header web.str(@crlf) ' which includes some information about the server and file web.str(string("Content-Length: ")) web.dec(size) web.str(@crlf) web.str(string("Content-Type: ")) web.str(type) web.str(@crlf) web.str(string("Connection: close")) web.str(@crlf) web.str(@crlf) ' This is the end of the HeaderYes, I agree. Without the TCP flow control VLC always stopped after about 1 second (although no packets were lost?!?).
There is something else: Downloads are now working? Last time I tried (without flow control) the computer always waited for the download to begin, but the server was already sending the file.
*sigh*
I can't wait until the next generation of propeller chips is available.
8 times more bandwidth...
This will be my first project with the Propeller and PropTCP is the reason I am taking the plunge (from the trusty ATMega168).··
Is there a link to a circuit diagram showing the Propeller chip (40pin DIP standalone) and the ENC28J60 together to the Network jack?· I'd like to assemble all the parts I'll be needing together and make a start.
Many thanks,
Anding