PDA

View Full Version : PropTCP Beta - Now fully MIT Licensed (w/ AJAX Enabled HTTP Server Example!)



Harrison.
12-08-2009, 03:12 PM
I would like to announce that PropTCP is now completely MIT licensed. I have decided that it's in the best interest of the Propeller community that all code be released under the MIT license. GPL compliance was impossible to enforce anyways, especially for small projects like PropTCP (violators probably wouldn't have cared about licensing anyways).

PropTCP now ships with a simple HTTP server example which only requires an ENC28J60 interface (configured for the standard YBOX2 hardware). The server supports 3 'simultaneous' connections, although it can only actually service one request at a time (the other requests are queued). The HTTP server serves up an AJAX enabled page which constantly displays a random number from Propeller. It also supports downloading of the Propeller HUB ram for speed test purposes. I am currently working on porting an older HTTP server example which serves files off of a SD card (it also has a 'full' featured file manager for file uploads and downloads).

I have attached my latest version. It is still in beta, but I am actively testing it on multiple Propeller boards to ensure that it is stable before I post it to the OBEX. I would appreciate any feedback anyone has.


There have been many updates to PropTCP over the past year. Below is a summary of changes:



+ Flexible Buffers - Buffer arrays are now supplied by the user application, which allows for different buffer sizes per socket. Buffer sizes must still be powers of 2.
+ TCP Retransmit - The sockets layer now performs retransmission whenever a packet is lost.
This results in very reliable communications. Unfortunately this also decreases thruput on high latency links.
+ RST Timeout - All socket states now have a timeout to prevent lockup issues.
+ Thread Safety - Public methods are now all thread safe. This means you can call them from different cogs without the previous lockup and corruption issues.
+ Handle Validation - Socket handles are now validated using socket state information.
+ Improved Error Codes - Better error codes are now returned.
+ Speed Improvements - The stack can now transmit data at up to 70KBytes/sec. This is dependent on link latency and buffer sizes.
+ Smaller Code - I refactored a bunch of code to improve efficiency. This in turn reduced compiled code size, and increased speed by roughly 15%.
+ MIT License - The code is now MIT licensed for OBEX compatibility!




Here's a short listing of examples to-be-released:



+ HTTP Bootloader - Program your Propeller over the internet (similar to Darco's YBOX2 bootloader, but hardware independent)
+ HTTP Webserver - A webserver with a SD card filesystem, web based file manager, and user loadable SPIN binary applications (should be better than the Parallax PINK...hopefully)
+ DNS Resolution - Resolve domain names (using TCP-based DNS queries w/ OpenDNS)


Post Edited (Harrison.) : 12/8/2009 10:07:04 AM GMT

Cluso99
12-08-2009, 04:51 PM
Congratulations Harrison. A big milestone indeed :)

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:

· Home of the MultiBladeProps: TriBlade (http://forums.parallax.com/showthread.php?p=786418),·RamBlade (http://forums.parallax.com/showthread.php?p=849265),·SixBlade (http://forums.parallax.com/showthread.php?p=780033), website (http://bluemagic.biz/cluso.htm)
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator) (http://forums.parallax.com/showthread.php?p=790917)
· Prop Tools under Development or Completed (Index) (http://forums.parallax.com/showthread.php?p=753439)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) (http://forums.parallax.com/showthread.php?p=778427) ZiCog (Z80) (http://forums.parallax.com/showthread.php?p=788511) , MoCog (6809) (http://forums.parallax.com/showthread.php?p=811043)
· Search the Propeller forums (http://search.parallax.com/search?site=parallax&client=parallax&output=xml_no_dtd&proxystylesheet=parallax&proxycustom=<HOME/>&ie=&oe=&lr=)·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz (http://www.bluemagic.biz)·· MultiBladeProp is: www.bluemagic.biz/cluso.htm (http://www.bluemagic.biz/cluso.htm)

Rayman
12-08-2009, 08:37 PM
I've got to try setting up a web server with this... Thanks, Harrison! (I try to avoid GPL like the plague...)

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

BR
12-08-2009, 08:42 PM
Oh, jeez. Now I've GOTTA get off my rear and put that YBOX2 kit together...

Thanks for this, Harrison.

Toby Seckshund
12-08-2009, 10:37 PM
I am trying to hold my concentration, to complete the dracBlade ( I still have my ENC plug-in board for bread boarding so with my "Lunchbox demoboard" I can have a Ybox2 in 10 minutes.

It was this time, last year, that Ethernet and the AVRs pointed me to the Ybox2 and hence the Prop. There will be so many projects for the Xmas break

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Style and grace : Nil point

jazzed
12-08-2009, 10:47 PM
Thanks Harrison. Do you have a DHCP example?

Bill Henning
12-09-2009, 06:59 AM
Thank you Harrison for MIT'ing your PropTCP!

I just took a quick glance at the driver, and I have a couple of questions:

1) You require a pin for INT, but it does not seem to be used anywhere... must it be hooked up?

2) You provide a 25MHz clock to ENCJ, what changes, if any, are needed if a 25MHz crystal is provided for the ENCJ? (other than removing the prop generating the 25MHz clock)

I am trying to see if I can use your stack with just four pins connected: /CS, CLK, DO, DI

Best Regards,

Bill


Harrison. said...
I would like to announce that PropTCP is now completely MIT licensed. I have decided that it's in the best interest of the Propeller community that all code be released under the MIT license. GPL compliance was impossible to enforce anyways, especially for small projects like PropTCP (violators probably wouldn't have cared about licensing anyways).

PropTCP now ships with a simple HTTP server example which only requires an ENC28J60 interface (configured for the standard YBOX2 hardware). The server supports 3 'simultaneous' connections, although it can only actually service one request at a time (the other requests are queued). The HTTP server serves up an AJAX enabled page which constantly displays a random number from Propeller. It also supports downloading of the Propeller HUB ram for speed test purposes. I am currently working on porting an older HTTP server example which serves files off of a SD card (it also has a 'full' featured file manager for file uploads and downloads).

I have attached my latest version. It is still in beta, but I am actively testing it on multiple Propeller boards to ensure that it is stable before I post it to the OBEX. I would appreciate any feedback anyone has.


There have been many updates to PropTCP over the past year. Below is a summary of changes:



+ Flexible Buffers - Buffer arrays are now supplied by the user application, which allows for different buffer sizes per socket. Buffer sizes must still be powers of 2.
+ TCP Retransmit - The sockets layer now performs retransmission whenever a packet is lost.
This results in very reliable communications. Unfortunately this also decreases thruput on high latency links.
+ RST Timeout - All socket states now have a timeout to prevent lockup issues.
+ Thread Safety - Public methods are now all thread safe. This means you can call them from different cogs without the previous lockup and corruption issues.
+ Handle Validation - Socket handles are now validated using socket state information.
+ Improved Error Codes - Better error codes are now returned.
+ Speed Improvements - The stack can now transmit data at up to 70KBytes/sec. This is dependent on link latency and buffer sizes.
+ Smaller Code - I refactored a bunch of code to improve efficiency. This in turn reduced compiled code size, and increased speed by roughly 15%.
+ MIT License - The code is now MIT licensed for OBEX compatibility!




Here's a short listing of examples to-be-released:



+ HTTP Bootloader - Program your Propeller over the internet (similar to Darco's YBOX2 bootloader, but hardware independent)
+ HTTP Webserver - A webserver with a SD card filesystem, web based file manager, and user loadable SPIN binary applications (should be better than the Parallax PINK...hopefully)
+ DNS Resolution - Resolve domain names (using TCP-based DNS queries w/ OpenDNS)



▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com (http://www.mikronauts.com) Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheus (http://mikronauts.com/products/morpheus/)dual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus (http://mikronauts.com/products/propteus/) and Proteus (http://mikronauts.com/products/propteus/) for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz (http://mikronauts.com/products/mikronauts-625mhz-crystal/)
Las (http://mikronauts.com/software-products/largos/) - Large model assembler for the Propeller Largos (http://mikronauts.com/software-products/largos/) - a feature full nano operating system for the Propeller

Dr_Acula
12-09-2009, 07:28 AM
This is very impressive! Thanks for sharing.

I have used some Lantronix ethernet to serial devices in the past and certainly it has been possible for people on the other side of the world to log in from a text based terminal program. But they only come up for a good price intermittently and the price varies a lot ($5 to $200 depending on the day). Something Prop based would be much better.

I've used DynDNS to link a fixed address to my changing home address. Originally I had it working as a small progam running on the background on a PC but more recently I had a browse through my router settings and DynDNS was in the router, so I just programmed the address into my router and I can get a 'business' grade permanent address for home prices. Very handy if you want to put a board on the net for home automation etc and not have it change address every day.

Harrison, can you expand a bit more on what you are doing, as I think it might be brilliant.

Can I ping your board?
Can I log into your board, ie I send a few nominal bytes and it sends me back something like 'hello, you are now logged in'

If you can do that, you could replace the Lantronix ethernet to serial device. Or even better, incoroporate it into the myriad of single board prop computers.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller (http://www.smarthome.viviti.com/propeller)

heater
12-09-2009, 07:36 AM
Dr_A: "Or even better, incoroporate it into the myriad of single board prop computers"

Wouldn't that be great! NetDracBlade.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.

Harrison.
12-09-2009, 08:48 AM
Thanks for the kind words everyone. Below are my answers to the questions posted above.


Bill Henning said...
1) You require a pin for INT, but it does not seem to be used anywhere... must it be hooked up?
The INT (interrupt) pin isn't being used right now. Just pass in -1 (or any other number) for now. I will likely remove that argument before I post the stack to the OBEX.


Bill Henning said...
2) You provide a 25MHz clock to ENCJ, what changes, if any, are needed if a 25MHz crystal is provided for the ENCJ? (other than removing the prop generating the 25MHz clock)
Pass in -1 for the xtalout pin number if you don't need the 25MHz clock. No other hardware or software changes need to be made.


Bill Henning said...
I am trying to see if I can use your stack with just four pins connected: /CS, CLK, DO, DI
Yeap, those are the only pins that need to be connected if you don't need the 25MHz Prop generated clock. Most of my boards only use those four standard SPI pins.



Dr_Acula said...
Can I ping your board?
Can I log into your board, ie I send a few nominal bytes and it sends me back something like 'hello, you are now logged in'
The stack doesn't currently support icmp ping, but that isn't difficult to add. I could probably whip up a quick demo running http and telnet. In fact, one of the earliest tests I did back in 2007 was to run Donglebasic (serial version of Femtobasic) over TCP w/ telnet.



jazzed said...
Thanks Harrison. Do you have a DHCP example?
Unfortunately I don't have any DHCP code yet. I can't figure out a good way of integrating it without using massive amounts of code space. If you really need DHCP then you might want to take a look at Darco's DHCP code for the YBOX2.

Bill Henning
12-09-2009, 11:36 AM
Thanks for the replies!

I've glanced over the rest of the files - very nice, clear to understand code.

A couple of possible memory saving suggestion:

- reuse the area used by the ENCJ initialization for buffer space
- reuse the area for the cog pasm code for more buffer space

There is also an object overlay loader somewhere in this forum... it would make it possible to dynamically load a dhcp object to obtain an IP address, then discard it. The same mechanism could be used to start the SPI cog, and to initialize the ENC28J60.

I really like the simplicity and the low cost of the ENC28J60 versus the WIZ5100 - and the ENC28J60 is readily available as a DIP part, a big plus in my opinion.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com (http://www.mikronauts.com) Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheus (http://mikronauts.com/products/morpheus/)dual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus (http://mikronauts.com/products/propteus/) and Proteus (http://mikronauts.com/products/propteus/) for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz (http://mikronauts.com/products/mikronauts-625mhz-crystal/)
Las (http://mikronauts.com/software-products/largos/) - Large model assembler for the Propeller Largos (http://mikronauts.com/software-products/largos/) - a feature full nano operating system for the Propeller

Post Edited (Bill Henning) : 12/9/2009 4:43:02 AM GMT

T Chap
12-10-2009, 10:22 AM
Thanks Harrison for posting this. I have been wanting to experiment with this for a while. Seems like a lot of fun.

Patrick1ab
12-10-2009, 05:12 PM
Hi Harrison!

I was also working on a simple http server:
http://forums.parallax.com/forums/default.aspx?f=25&m=406015

The structure of my code isn't very nice and uses a lot of memory due to the huge bufferspace, but it's pretty fast:
90 kBytes/s @ 80 MHz Propeller clock frequency
106 kBytes/s @ 100 MHz

Toby Seckshund
12-11-2009, 06:53 AM
You can clock the Prop from the ENC, rather than the other way round. At wake up the ENC puts out 25MHz/4 = 6.25MHz, just right for 100MHz with a pllx16. Or the output divider could be set to /2 and
give 12.5MHz with pllx8.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Style and grace : Nil point

Patrick1ab
12-11-2009, 07:14 AM
I think I found a bug:

The dec function in the api_telnet_serial isn't working correctly, although the code of this function hasn't changed compared to previous versions of PropTCP.

I found out by chance, while I tried to modify my webserver to work with the newest version.

localroger
12-11-2009, 08:57 AM
Thanks, Harrison. This is a very positive thing for the Propeller.

I started working on a stack of my own mainly because of the license problem. I will probably finish it because with your changes our functionality has diverged pretty far too. I am using the Spin object model to do code memory optimization that would probably save about 4K of Hub RAM if I just copied your logic. (If you define an array of Socket objects, you can get rid of *all* the dereferencing e.g. socketdata[pkt][position_in_pkt] stuff. That saves an AMAZING amount of code Hub RAM.)

I also want to optimize mine to work right with lots of fast, short connections, because that's how I have to deal with some embedded controllers I use that I can't reprogram if I want decent data bandwidth. That's how I discovered the timeout problem, something that you might notice once every day or two in a normal application was showing up every five minutes for me because my app requires opening a connection, doing your stuff, closing it, and then repeating several times a second. With the default timeouts in the usual TCP stacks and no option for UDP in the pre-programmed other device, it's the only way to get throughput. But I'd be the first to admit it's a specialty app and what I'm going to do to make it work might not be anybody else's cup of tea.

Harrison.
12-11-2009, 09:54 AM
Patrick1ab said...
The structure of my code isn't very nice and uses a lot of memory due to the huge bufferspace, but it's pretty fast:
90 kBytes/s @ 80 MHz Propeller clock frequency
106 kBytes/s @ 100 MHz
You were probably using a very old version of the stack that didn't support TCP retransmit. Basically it just sent out packets blindly, as fast as it could go, which is why you saw pretty nice speeds. The latest version waits for ACKs after every packet (in fact, it sends a data packet and then an empty packet to force the remote side to ACK). Waiting for ACKs adds in a few ms of delay, which slows down the effective thruput. I think the benefit of having reliable transfers is worth the speed loss. I really wish the Propeller Tool supported conditional compilation, which would allow me to make TCP retransmit optional.


Patrick1ab said...
The dec function in the api_telnet_serial isn't working correctly, although the code of this function hasn't changed compared to previous versions of PropTCP.
Could you post or send me your code? My demo webserver included in the first post uses .dec() to send the random number. You could be forgetting to flush the transmit buffer before closing the connection.



Toby Seckshund said...
You can clock the Prop from the ENC, rather than the other way round. At wake up the ENC puts out 25MHz/4 = 6.25MHz, just right for 100MHz with a pllx16. Or the output divider could be set to /2 and
give 12.5MHz with pllx8.
That's a great idea! When I first wrote the stack (Dec 2006) the Propeller hadn't been tested at speeds above 80MHz. Seems like everyone is running the Propeller at 100MHz these days (in fact, I ordered ten 6.25MHz xtals last night).



localroger said...
I started working on a stack of my own mainly because of the license problem. I will probably finish it because with your changes our functionality has diverged pretty far too. I am using the Spin object model to do code memory optimization that would probably save about 4K of Hub RAM if I just copied your logic. (If you define an array of Socket objects, you can get rid of *all* the dereferencing e.g. socketdata[pkt][position_in_pkt] stuff. That saves an AMAZING amount of code Hub RAM.)
Very Interesting. The beta release posted in the first post switched from using massive pointer dereferencing to using direct arrays. I moved the packet array into the socket layer, and switched to using separate arrays for all the socket state variables. This saved a couple hundred longs (while increasing speed a bunch), but I don't think it saved as much as 1K longs. I also no longer have to do complicated pointer math to get to the socket state variables.


localroger said...
I also want to optimize mine to work right with lots of fast, short connections, because that's how I have to deal with some embedded controllers I use that I can't reprogram if I want decent data bandwidth. That's how I discovered the timeout problem, something that you might notice once every day or two in a normal application was showing up every five minutes for me because my app requires opening a connection, doing your stuff, closing it, and then repeating several times a second. With the default timeouts in the usual TCP stacks and no option for UDP in the pre-programmed other device, it's the only way to get throughput. But I'd be the first to admit it's a specialty app and what I'm going to do to make it work might not be anybody else's cup of tea.
I ran into those connection issues about 8 months ago when I was working on a project that used a bunch of small short term connections. It became more apparent over high latency / lossy networks (like wireless). I had to add a bunch of timeouts and retransmit code to make communication reliable. The current beta should be very resilient to connection problems. I use it in a couple of mutli-cog projects, with long term and short term connections and haven't seen any issues yet.

Post Edited (Harrison.) : 12/11/2009 3:00:54 AM GMT

localroger
12-11-2009, 09:34 PM
Harrison: The beta release posted in the first post switched from using massive pointer dereferencing to using direct arrays. I moved the packet array into the socket layer, and switched to using separate arrays for all the socket state variables. This saved a couple hundred longs (while increasing speed a bunch), but I don't think it saved as much as 1K longs. I also no longer have to do complicated pointer math to get to the socket state variables.

You might want to look into this trick since it saves even more speed and RAM; the principle is that you create a socket object with its own data and code, and in the obj block declare an array of them:



con
max_sockets = 5

obj
nic := "driver_enc28j60"
sck[max_sockets] : "socket"

As elsewhere, the different instances of socket.spin share DAT but have their own VAR areas.· This means that socket.spin code doesn't need to use arrays at all -- the socket index is implicit in the function call...



sck[handle].tcpmsg

This way, within the tcpmsg routine you don't need to use array references at all, just direct memory variables.· I also added functions in the nic object to read and write common data structures to the packet buffer and SRAM without dereferencing to [pkt] every time.



var
byte state
long srcip
long seqnum
long acknum
'... etc.

obj
nic := "driver_enc28j60"

pub tcpmsg
case state
sck_closed: return
sck_listen:
if (hdrflags & tcp_syn)
srcip := nic.readpktlong(ip_srcaddr)
srcport := nic.readpktword(ip_srcport)
'...and so on

Of course you need to provide functions for sck to report its state and such to the parent object, and the nic object must be available to both the parent and socket (which is no problem since it uses DAT storage exclusively).· In code blocks where I was just duplicating your functionality this was consistently shrinking it to about 60% of its original size.

Harrison.
12-12-2009, 02:32 AM
localroger,
I'll definitely have to try that. I've always thought that function calls were much more expensive since they have to push stuff onto the stack. I always assumed that using arrays directly were much faster since it probably doesn't need to do extra stack operations. I guess my reasoning was that it's better to have faster thruput than to have smaller code. Maybe my real issue is I need to compile with BST and look at the bytecode generated.

EDIT: Ah, I think I see what you are talking about now. You moved a lot of the state machine code into the Socket object, which allows direct variable accesses without expensive function calls / array dereferencing inside the state machine code. This sounds much better than the current method I'm using.

Post Edited (Harrison.) : 12/11/2009 7:40:44 PM GMT

localroger
12-13-2009, 08:55 AM
Harrison, if you're willing to refactor using my techniques it's a total waste of time for me to recode the object. My fundamental objective is reducing Hub RAM use. This is imperative if you're also including FSRW or some substitute for it in a project, and you want to have business logic on top of all that.

Bill Henning
01-05-2010, 04:24 AM
Ok, I am finally trying to get ethernet running on my props :)

I built a Ybox2, and a board of my design - the idea was that I can use the Ybox2 (known good design) to help debug my board.

Harrison or localroger - would you guys have any tcp stack debugging code kicking around that might help me?

Here's where I am at:

I downloaded the latest PropTCP beta from the first post in this thread, and made the following changes:




mac_addr byte $02, $01, $02, $03, $00, $05 ' device mac address, must be unique

ip_addr byte 192, 168, 1, 8 ' device's ip address
ip_subnet byte 255, 255, 255, 0 ' network subnet
ip_gateway byte 192, 168, 1, 254 ' network gateway (router)
ip_dns byte 192, 168, 1, 254 ' network dns




I am trying to get the web server demo running.

On a Ybox 2, I start with:




{ybox2} sock.start(1, 2, 3, 4, -1, -1, @mac_addr, @ip_addr)




Maybe I did a bad job of soldering, because I don't even get a link light on my switch!

On my board, I start with:




{other} sock.start(23, 20, 21, 22, -1, -1, @mac_addr, @ip_addr)




Here at least I get a link light!

I'm now going back to debugging...

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com (http://www.mikronauts.com) Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheus (http://mikronauts.com/products/morpheus/)dual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus (http://mikronauts.com/products/propteus/) and Proteus (http://mikronauts.com/products/propteus/) for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz (http://mikronauts.com/products/mikronauts-625mhz-crystal/)
Las (http://mikronauts.com/software-products/largos/) - Large model assembler for the Propeller Largos (http://mikronauts.com/software-products/largos/) - a feature full nano operating system for the Propeller

Harrison.
01-05-2010, 04:28 AM
The example_httpserver.spin object included in the zip file is built to run unmodified on a ybox2. The ybox2 hardware requires that the prop generate the 25MHz clock for the enc28j60, so you would want to run with the following (this is what example_httpserver.spin is pre-configured with):


sock.start(1, 2, 3, 4, -1, 7, @mac_addr, @ip_addr)

Bill Henning
01-05-2010, 04:30 AM
Thanks - will try that right now!

I forgot to restore the 7 when it did not work on my board and I added back in the alternate start line for Ybox2.

(WOW that was a fast response!)


Harrison. said...
The example_httpserver.spin object included in the zip file is built to run unmodified on a ybox2. The ybox2 hardware requires that the prop generate the 25MHz clock for the enc28j60, so you would want to run with the following (this is what example_httpserver.spin is pre-configured with):


sock.start(1, 2, 3, 4, -1, 7, @mac_addr, @ip_addr)


▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com (http://www.mikronauts.com) Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheus (http://mikronauts.com/products/morpheus/)dual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus (http://mikronauts.com/products/propteus/) and Proteus (http://mikronauts.com/products/propteus/) for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz (http://mikronauts.com/products/mikronauts-625mhz-crystal/)
Las (http://mikronauts.com/software-products/largos/) - Large model assembler for the Propeller Largos (http://mikronauts.com/software-products/largos/) - a feature full nano operating system for the Propeller

Bill Henning
01-05-2010, 04:34 AM
That did it.. IT WORKS ON YBOX2!!!!!!!!!!!!

Now I can compare it to my setup and find out where the problem is...

Thanks!


Harrison. said...
The example_httpserver.spin object included in the zip file is built to run unmodified on a ybox2. The ybox2 hardware requires that the prop generate the 25MHz clock for the enc28j60, so you would want to run with the following (this is what example_httpserver.spin is pre-configured with):


sock.start(1, 2, 3, 4, -1, 7, @mac_addr, @ip_addr)


▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com (http://www.mikronauts.com) Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Morpheus (http://mikronauts.com/products/morpheus/)dual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory IO board kit $89.95, both kits $189.95
Propteus (http://mikronauts.com/products/propteus/) and Proteus (http://mikronauts.com/products/propteus/) for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz (http://mikronauts.com/products/mikronauts-625mhz-crystal/)
Las (http://mikronauts.com/software-products/largos/) - Large model assembler for the Propeller Largos (http://mikronauts.com/software-products/largos/) - a feature full nano operating system for the Propeller

Patrick1ab
01-08-2010, 08:15 AM
Hi Harrison!

I tried to work on my web server again, but I still can't find the bug.

It works fine, if the data that is sent back to my computer consists only of a few bytes:


HTTP/1.0 200 OK

<html>Liste der Dateien auf dem Webserver:
------------------------------------

DRWHO11.AVI
INDEX.HTM
INTHEEND.MP3
LICENCE.TXT
PRAKTI.JPG
PROBE.TXT
SCHUH1.M4A
SCHUH4.M4A
SCHUH13.M4A
SCHUH16.M4A
T-HOME.MP3
36GRAD.MP3
BOOTEX.LOG
CONFIG.HTM
DAVINCI.M4A
SECURITY.HTM
PIRATE.MP3

Dateien insgesamt: 17</html>


As soon as bigger files are being transmitted, this is what happens (first line is the reply of Fiddler2):

HTTP/1.0 200 This buggy server did not return headers

HTTP/1.0 200 OK
Server: PropServer
dy bgcolor="000000">
<head>
<font color="65FF85" size="10">
Testseite
</font>
</he<html>
<title>Testseite</title>
<body bgcolor="000000">
<head>
<font color="65FF85" size="10">
Testseite
</font>
</head>
<font color="65FF65">


Das ist eine Testdatei mit der ich überprüfen möchte, ob die Übertragung von Daten der SD-Karte [...]


As you will see below, the header is corrupted and parts of the file are being send twice. It looks like a memory management problem, but how can that be possible?

This is an excerpt of the code:



if sdfat.popen(@fname,"r") <> 0
web.str(@http404)
else
web.str(@http200)
web.str(string("Server: PropServer"))
web.str(@crlf)
web.str(string("Content-Length: "))
web.dec(size)
web.str(@crlf)
web.str(string("Content-Language: de"))
web.str(@crlf)
web.str(string("Content-Type: "))
web.str(type)
web.str(@crlf)
web.str(string("Connection: close"))
web.str(@crlf)
web.str(@crlf)
if m==1
return 0
else
repeat
h:=sdfat.pread(@sendwebBuffer,BufferSize)
if h=<0
quit
else
if h<BufferSize
repeat n from h to (BufferSize-1)
sendwebBuffer[n]:=32
sendwebBuffer[BufferSize]:=0
web.txdata(@sendwebBuffer,h)
return 0




Basically this code is doing the following things:
1. Try to open the file; if it's not found, send an error message
2. else send the header (if m==1, the request was "HEAD")
3. send the file in pieces of BufferSize bytes (I set BufferSize to 4096) to the buffer

The subroutines "webserver" and "_webReadLine" are nearly the same ones you are using, except that I only allow one socket.

I can post the full code too, but at first I have to translate my comments into English http://forums.parallax.com/images/smilies/wink.gif

Harrison.
01-08-2010, 09:44 AM
Could you post the entire archive zip? That way I can just try running it on my board for debugging.

Patrick1ab
01-09-2010, 06:28 AM
In the attachment you will find the archive containing all the files needed to run the HTTP server.
I know that the code of the main program looks a bit ugly, but I'm going to seperate it to several procedures soon.

Thanks for your help!

Harrison.
01-09-2010, 09:17 AM
Looks like there was a bit of confusion with how to use the buffers. The new tx/rx buffer parameters for the .connect() and .listen() functions are the dedicated circular buffers to be used by the stack. You should not touch these buffer arrays after you pass them to the .connect() or .listen() functions.

I've attached the modified version that should work very well. I had to make a few changes to make it work. The changes were:
- Added new circular buffer arrays for the .listen() call. These have to be reserved for the tcp stack, and cannot be touched by your application.
- Disabled sending of Content-Length header (it was sending 0 length because size was not set). This confused my browser (Firefox 3).
- Optimized the sd -> tcp stack transfer repeat loop. You had some unnecessary code to clean the array.

Other than that, the code seems to work very well. I can download files from an old 32MB SD card at about 125Kbytes/sec with a 100MHz Propeller clock. I tested MP3 streaming with VLC, and it works very well (even supports stopping and pausing correctly!). I guess the stack's new TCP flow control stuff works great for streaming, especially since VLC only requests a bit of data at a time.

Sorry for not making the usage of these new parameters clearer. It's my fault that you had to wrestle with this problem.

Patrick1ab
01-09-2010, 08:13 PM
Wow, thanks for your great work, Harrison!
You have achieved this in a few hours, what I couldn't in several weeks http://forums.parallax.com/images/smilies/shocked.gif


Harrison. said...

- Disabled sending of Content-Length header (it was sending 0 length because size was not set). This confused my browser (Firefox 3).



Oops, sorry... It seems I've deleted something I commented out before due to debugging. I'm going to correct that immediately.
It should work again, if you insert this line

size:=sdfat.get_filesize

after the "else":



if sdfat.popen(@fname,"r") <> 0 ' Try to open the requested file
web.str(@http404) ' if it is not found on the sd card send back the 404 error message
else
size:=sdfat.get_filesize
web.str(@http200) ' file has been found and will be transmitted,
web.str(string("Server: PropServer")) ' but first we are going to send the header
web.str(@crlf) ' which includes some information about the server and file
web.str(string("Content-Length: "))
web.dec(size)
web.str(@crlf)
web.str(string("Content-Type: "))
web.str(type)
web.str(@crlf)
web.str(string("Connection: close"))
web.str(@crlf)
web.str(@crlf) ' This is the end of the Header





Harrison. said...
I tested MP3 streaming with VLC, and it works very well (even supports stopping and pausing correctly!). I guess the stack's new TCP flow control stuff works great for streaming, especially since VLC only requests a bit of data at a time.


Yes, I agree. Without the TCP flow control VLC always stopped after about 1 second (although no packets were lost?!?).
There is something else: Downloads are now working? Last time I tried (without flow control) the computer always waited for the download to begin, but the server was already sending the file.

Patrick1ab
01-13-2010, 07:34 AM
Hmm, DivX content isn't playing continuously. http://forums.parallax.com/images/smilies/sad.gif

*sigh*

I can't wait until the next generation of propeller chips is available.
8 times more bandwidth...

Anding
01-23-2010, 12:32 PM
Dear Harrison and other friends,

This will be my first project with the Propeller and PropTCP is the reason I am taking the plunge (from the trusty ATMega168).··

Is there a link to a circuit diagram showing the Propeller chip (40pin DIP standalone) and the ENC28J60 together to the Network jack?· I'd like to assemble all the parts I'll be needing together and make a start.

Many thanks,

Anding

Bill Henning
01-23-2010, 12:38 PM
I recommend that you take a look at the Ybox2 at

www.ladyada.net/make/ybox2/ (http://www.ladyada.net/make/ybox2/)

I've built two, and they are a great way of getting starterd with PropTCP!


Anding said...
Dear Harrison and other friends,

This will be my first project with the Propeller and PropTCP is the reason I am taking the plunge (from the trusty ATMega168).

Is there a link to a circuit diagram showing the Propeller chip (40pin DIP standalone) and the ENC28J60 together to the Network jack? I'd like to assemble all the parts I'll be needing together and make a start.

Many thanks,

Anding

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com (http://www.mikronauts.com) E-mail: mikronauts _at_ gmail _dot_ com 5.0" VGA LCD in stock! (http://forums.parallax.com/showthread.php?p=852195)
Morpheus (http://mikronauts.com/products/morpheus/) dual Prop SBC w/ 512KB kit $119.95, Mem+2MB memory/IO kit $89.95, both kits $189.95 SerPlug $9.95 (http://mikronauts.com/products/serplug/)
Propteus (http://mikronauts.com/products/propteus/) and Proteus (http://mikronauts.com/products/proteus/) for Propeller prototyping 6.250MHz custom Crystals run Propellers at 100MHz (http://mikronauts.com/products/mikronauts-625mhz-crystal/)
Las (http://mikronauts.com/software-products/largos/) - Large model assembler Largos (http://mikronauts.com/software-products/largos/) - upcoming nano operating system

Anding
01-23-2010, 01:09 PM
Thanks Bill. The Ybox2 looks like a fine idea. I will order a kit and a programming cable from the same supplier.

- Any other (inexpensive) parts / accessories handy for small Propeller projects while I am at it with Adafruit?

- I guess PropTCP is not the default software on the PropTCP? Would it be better if I am to switch over to PropTCP once I've built the kit and have it working?

Toby Seckshund
01-23-2010, 04:42 PM
@ Anding

A year ago I was cursing all my efforts on a ATMEGA168, 32, 644 using "Bens" Bascom code and spotted the YBOX2. My homemade attempt worked first time. I should get around to writing/modding some more useful "Widgets".

I think the software is all down to Darko

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Style and grace : Nil point

Szabi
01-29-2010, 05:32 AM
Harrison,



you did a really nice job! Thanks for that!

I just started to learn the propeller and spin language...I just simply love it!·Now that I start understanding things I decided to play with your driver :-) Since I'm just a beginner and·not really familiar with html code an java script I would need your help. Could you or anybody else post some sample spin codes, for example how I could read returned values from a webppage into variables? For example let's create kind of textbox on a page along with a button·and by clicking the button·the string·should be placed into a variable (that later I can display on the tv for ex.)

Another sample I would need is for·example how I couls send and receive some strings through TCP on any port (or better on a different port but running·paralelly the webserver)·For example I would send a string on port 23 with telnet and that soould be displayed on the tv and echoed back to the telnet client ...or something like this.

Simply any examples would be welcome I could play with :-)



Many thanks in advance!

Tired2
02-11-2010, 07:21 AM
Szabi,

I am basically attempting the same thing as you, and also pretty new to the propeller.

From what have found, PropTCP by Harrison is utilized by the ybox2, so a 'quick and easy' way to get started is by purchasing a ybox2 kit and getting it going.

I am pretty poor currently, so I am bread boarding out my own ybox2 using an ENC28J60 breakout board I found at sparkfun for $35. Granted, this is almost half the price of the $75 ybox2 kit, but I already had the propeller, propplug, etc needed to get it going, and I plan to reuse the board in my actual project down the road a bit.

From what I can tell, a good starting place for being able to control things like LEDs through web addresses is to start with wwwexample.spin in the ybox2 svn repository.

Once you get it flashing an LED, you can break down how it achieves it, and go from there. The example seems to have it all, though unfortunately I have to go now prior to setting up my network to test with :(

Tired2
02-25-2010, 04:59 AM
Harrison,

I have put many hours into playing with "example_httpserver.spin" included in your proptcp 5.0-7beta, and I have had some mixed results.

I want to use this for a project that I clearly do not have the skillset for yet, so I am basically hoping to get a few more bits of info from you that might help me pull everything together.

I am having three main problems...

First,

I am unable to find a way to pull arguments from cgi calls. I see that the 'args' are parsed out of @reqstr, but I'm not sure how to get those into a string I can process for myself.

I am hoping to build an internet enabled hot tub controller, but the framework for it should be like most any other web-based control apps. I'd like to be able to host a simple page of links to something like http://<prop ip>/tempset.cgi?temp=100. I also would like to display current variables similar to what is done with the random number, which should be easy if I can get the ajax figured out.

That brings me to my second problem. I am able to use your if/elseif structure to run my own function based on my own custom cgi scripts, I.E.: http://<prop ip>/ledblue.cgi




........
elseif str.indexOf(@reqstr, string("ledblue.cgi")) <> -1 ' if GET contains "ledblue.cgi"
Sound(1, 200, 25_000_000, 50_000) ' sound from piezo speaker
GlobalVar := 100 ' set a global variable to value 100
sock[sockidx].str(string("<html><body><B>Sound Played</B><BR><BR></body></html>")) ' output confirmation page
....... etc




My problem is that the "Sound" Function works, and I can then use that to set global variables, but I I cannot for instance turn on an LED with: outa[ledpin]:=1 (where ledpin is a CON, or even a set pin number) Any other functions with pin I/O seem to fail also... Ideas on why? Does it have to do with it being PRI vs. PUB? I'm lost.

I can also work around that problem by flagging global variables and having my main functions watch those instead, but that is not how I'd like it to work, and I fear that my workaround will cause a future problem.

Okay, if you are still with me at this point, my third problem is with AJAX...

Do you have a less "trimmed" down version of "ajax.js" you could post? Maybe with comments? I have tried everything I can think of to have an additional variable (temperature) updated alongside the random number, but it always propagates the tags with whichever value I feed it first in the HTML file. They both update dynamically, but they are the same value for some reason. I think it is the way ajax() handles inputs, but it is hard to tell for sure, being completely new to the concept.

I think with just a slight extension of example_httpserver.spin, many people of my skillset could make some really cool network appliances with control options of computer, iphone, android phone, and anything in between.

Thanks for all your hard work on proptcp, the example is just so good that I want to push the limits of my project to what I know proptcp can do for me, it is just hard for newbies to jump in.

Harrison, let me know if you can help out, or anyone else who might know what I'm doing wrong.

Thanks.

Post Edited (Tired2) : 2/24/2010 10:08:51 PM GMT

Harrison.
02-25-2010, 01:57 PM
Your pin output problem is likely caused by incorrect I/O direction settings for the webserver cog. Each cog has its own I/O direction registers so you will have to make sure you set dira correctly for each cog.

An easy way to 'fix' this problem would be to set dira at the same time you set the pin state to high or low. The issue with doing this is that you will have pin output conflicts between the separate webserver cogs. Once an output pin is set to high, no other cog can set it to low. Information on how the I/O pins are shared between cogs can be found in the Propeller Manual.

You will have to manually process the GET parameters for now. The example_httpserver.spin object is extremely simple and was provided to show how to write a super simple webserver for the Propeller. The AJAX javascript provided with it is also super simple and only supports a single request at a time (it has an internal 'lock' to prevent multiple requests at the same time). Below is the source code for the AJAX stuff, which should be pretty easy to modify to support multiple requests (although that could overload the Propeller).



// ref: http://andreaslagerkvist.com/archives/2009/06/12/super-simple-ajax/
// orig src: http://aframework.googlecode.com/svn/trunk/__v3/aFramework/Modules/Base/superSimpleAjax.js
// compress: http://dean.edwards.name/packer/

var ajaxBusy = false;

function ajax(url, updateID, repeatDelay) {

if (ajaxBusy) {
// busy with previous request, don't do anything!
return;
}

ajaxBusy = true;

// Create ajax request object
var requestObject;

try {
requestObject = new XMLHttpRequest();
}
catch (e) {
requestObject = new ActiveXObject('Microsoft.XMLHTTP');
}

// This runs when request is complete
var onReadyStateChange = function () {
if (requestObject.readyState == 4) {
if (updateID) {
document.getElementById(updateID).innerHTML = requestObject.responseText;
}
ajaxBusy = false;
if (repeatDelay > 0) {
setTimeout('ajax(\'' + url + '\',\'' + updateID + '\',' + repeatDelay + ')', repeatDelay);
}
}
};

requestObject.open('GET', url + '?' + (new Date()).getTime(), true);
requestObject.onreadystatechange = onReadyStateChange;
requestObject.send(null);
}

Tired2
02-25-2010, 10:18 PM
Harrison,

Thank you very much for the full ajax script. As for the pins, I knew there was something I was missing, I knew all cogs had access to I/O, but I did not realize they had different direction registers.

I'll see what I can put together.

Thinking about it... I doubt multiple sockets will have to modify the same I/O at the same time.. I mostly want extra sockets in case I leave a connection open on one device (computer), and want to access it from elsewhere with iphone. Luckily for this project, the time aspect is way slowed down compared to my last one... responses in 5 seconds should be about as fast as I need :) That is a welcome change. I think I'll go for the global variable route at first. I'm excited.

Post Edited (Tired2) : 2/25/2010 4:47:07 PM GMT

Tired2
02-26-2010, 11:39 PM
Okay,

After a few hours of playing around, I am pretty happy with the results so far.

I have the example_httpserver.spin set up with a /led.cgi page that allows for on/off control of the three colors of an RGB led.

I also chopped up the DAT entries to make the HTML code that is reused often only take a single DAT entry. There is an iphone specific piece of code in there as well that allows for custom scale on iphone... I'm no web developer, but I'm planning to hack away at this until I get something I like.

Here is a video:
www.youtube.com/watch?v=XNUQYdgvyI0 (http://www.youtube.com/watch?v=XNUQYdgvyI0)

Attached is my code if anyone cares to take a look, as well as the object info screenshot. It should run on stock ybox2 hardware, as the example_http.spin does in proptcp.

Of course, all of the work was pretty much done by Harrison, I just expanded the example to prove to myself I could make it control things from the web. Next on my list is to get argument parsing worked out so I can set values like on the ybox2 web example, i.e. /ledcolor.cgi?=FF0462.

After that, its on to ajax to see if I can get it to send requests and display multiple variables without refreshing the page.

Thanks Harrison!! My hot tub will be remote controlled in no time :) (or about 3 months)

davidsaunders
03-20-2011, 11:10 PM
Harrison:
Thank you. This will serve as the best documentation reference for TCP, IP, UDP I can find (as a refresher) anywhere on the net. I will be using it as a reference in my own Propeller assembly socket library.

There is an interesting bug in the module util_strings in the procedure toLower, this procedure assumes that all characters in the string are letters (upper or lower case), If you OR another character with hex $20 you will quite often get unwanted results. As a quick fix just do a quick bounds check on each character. Sorry can not give an example as I do not use SPIN enough to know how to construct a conditional statement off the top of my head.

malmeidap
11-20-2011, 08:03 PM
Hi Harrison:

Last week I bought the smallest ENC28J60's package from Hong Kong (http://www.ebay.com/itm/ws/eBayISAPI.dll?ViewItem&item=260747627291) to install it on the propeller demo board and test ur http web server example, but it didnt function. Can u help me pls ? i need the pin connection diagram for this package and the changes on the driver (if they exists) to test ur Web demo. The pin diagram of the element is published on the URL. Its very important help me pls :smile:

87031

Rayman
11-20-2011, 09:04 PM
Make sure you pull the CS and Reset pins high with a 10k resistor or so.

Cluso99
11-20-2011, 10:54 PM
Nice cheap module. Did you notice the logo on the pcb - Zero Research.

malmeidap
11-20-2011, 11:21 PM
Make sure you pull the CS and Reset pins high with a 10k resistor or so.

I connected it like ur diagram on these url http://code.google.com/p/proptcp/downloads/list.

* I dont know if the DIP´s 40 package r the same as the Q44 package.
* I read the driver, I saw that u generate an internal 25 MHz with the propeller systems clock, but these chip has his own 25 MHz clock. It can be a problem or not ?
* I found these on google http://ucontroller.com/documentation/PropNICDoc.html, r u agree with these information ?


Nice cheap module. Did you notice the logo on the pcb

No, I didnt see what do it have ?

Rayman
11-20-2011, 11:37 PM
I think it should just work... It's just a simple SPI connection...
Make sure you don't have the SPI SO and SI reversed...

BTW: I'm not Harrison, but I'm trying to help anyway...

The external crystal is fine.

I've made my own board for the DIP version of the ENC chip here:
http://www.rayslogic.com/Propeller/Products/PropBox/PropBox.htm
There's a schematic in case that helps.
I used Harrison's code and it just worked...

malmeidap
11-21-2011, 12:06 AM
BTW: I'm not Harrison, but I'm trying to help anyway...

Ty Rayman, sorry i was confused. Ur schematic its very helpful. I will try to use the mini ENC28j60 with the DIP version, then I will try with the demo board I will publish the solution as soon as possible ;) !!!

Toby Seckshund
11-21-2011, 08:15 AM
Just a thought, the ENC28j60 chip is quite thirsty on current so it might be worth keeping an eye on the power rail stability.

Rayman
11-21-2011, 10:39 AM
Actually, I just looked up my code and I think I had to make changes in MAC and IP address parts of the first DAT block.
Basically, give it an IP address that will work in your network.

Also, I had to change this line to tell it which pins I'm using:
' Init the TCP/IP driver
sock.start(0, 1, 2, 3, -1, 7, @mac_addr, @ip_addr)

yeti
11-21-2011, 01:34 PM
sock.start(0, 1, 2, 3, -1, 7, @mac_addr, @ip_addr)
A -1 instead of 7 as 6th parameter switches off generating 25MHz on P7.
...and pullups are not needed on /CS and /RESET cause the enc-chip has weak pullups inside (mentioned somewhere in the datasheet).