Basic GET and POST Object [Updated 1/1/2011]

Mike G · 2010-11-21 19:52

The attached is a mixture of stuff from Roy to Tim. I added simple parsing of the HTTP header so we can grab HTTP GET or POST name value pairs.

request.__GET(string("name"))

Still a little raw but it works. The __POST method needs some work... It has to do with finding the header body, 0x13, 0x13. [Fixed:] I was trying to match the pattern 0x13, 0x13 to find the request body. Changed the pattern to 0x13, 0x10, 0x13, 0x10 and that works great.

Updated (11/23/2010) a object a bit and cleaned up the code. Make sure you set your IP, MAC etc. I made a host file entry to resolve "spinneret", so make sure you edit the HTTP FORM node according to your setup.

This object parses the HTTP header and buffers the request URL, just in case you need to know the last page served. Since I'll be using XMP RPC, I determine the file type. If the type is "spin" then I'll run the appropriate service method.

AN SD card loaded with the the spinneret (attached) is required for this demo to function properly.

[11/26/2010]
spnneret.zip contains HTML pages

[11/27/2010]
Added a simple dispatch method to load dynamic content

[11/28/2010]
Added dynamic file generation

[12/12/2010]
Updated website files and added demo for client and server. Included __GET(string("name")) example in the Service method. Querystring values are dynamically added to the server.shp page. I've been using Firefox which works fine. IE has problems.

[12/19/2010]
Figured out the problem with IE... it was favicon.ico. IE looks for that file and I did not implement a file not found. Well, I still did not implement file not found but I did add an ico file to the web project. I also updated the HTTPRequest object so it parses the file extension and exposes the extension to the caller.

[12/23/2010]
Added IsPostBack method so I can determine if the page has posted back. I still need to add html encoding and decoding. I'm leaving the server up for a few hours today, 12/23/2010 from 10:00 to 2ish (GMT-7).
URL: http://spinneret.servebeer.com:5000/index.htm or http://spinneret.servebeer.com:5000

Lots of folks on the Propeller form helped test the Spinneret as such I had to make a few changes.

[1/1/2011]
Added URL decoding and page not found.

Source code:
http://www.agaverobotics.com/spinneret/source/

CassLan · 2010-11-26 14:56

I'm working with your code, and the GetEndPointer seems to give the end of the POST, and not the end of the QuerySTring value that your looking for. The GetStartPointer works well.

I think it has to do with looking for the termination character, and assuming that there will not be further POST data following.

Rick

Mike G · 2010-11-26 15:08

I refactored a few items, added Roy's "SD2.0_FATWrapper.spin" which includes Kyle's awesome "SD2.0_FATEngine.spin".

The HttpRequest object is setup to enumerate up to 10 sub directories. If the url request does not contain a file then index.htm is served up which is configurable.

I'm still testing, refactoring, and encapsulating but I thought this was a good spot to post my progress.

Mike

Mike G · 2010-11-26 15:13

@CassLan,

Thanks for the feedback. Taking a look at the pointer , I probably messed something up. I kinda' decided to hide the pointer stuff for a more user friendly object interface. The pointers change depending on the method that just ran which is not ideal.

CassLan · 2010-11-26 15:27

Thats cool Mike, but how do we extract the values we need from the POST Data without using those Pointer values?

Maybe I'm not using the object properly

Mike G · 2010-11-26 15:43

Paradigm shift... I'm exposing dynamic pointers while many of the obex object use zero terminators. I have to think about it a bit.

Mike G · 2010-11-27 05:57

Updated the HttpRequest object to execute the appropriate method if a dynamic page is requested. A dynamic page has the extension *.sph

SHP = Spinneret Host Page

If the request is service.shp, the DispatchProcess function checks to see if the request is a dynamic or static page. Static pages just pass on through to the StaticFileHandler. A dynamic page request is passed to MapMethod which executes the mapped method.

Request.IsMethod(string("service"))

There's not much going on in the service method. I just load a test page.

The next step is to process create a temporary file cache on the SD card so we can build a dynamic file on the fly and serve it up. I also have to do some work on the __POST and __GET methods as they probably don't work well with the new changes.

Once that's done, I'll add comments and clean up the code.

CassLan · 2010-11-27 07:58

Mike that sounds great, I've been a little reluctant to create any dynamic pages since I feel like I will be completely out of spec, when a spec is created.

Mike G · 2010-11-28 15:58

I fixed a lot of bugs in the original HttpRequest file. Mainly handling memory pointers. I kinda screwed that up, surprisingly that the object worked at all.

Anyway, I added the ability for dynamic files. This is the first iteration and it works so I'll be building from this point. The file size is limited to 2k right now.

All you have to do is add  to a server page. Server pages have the .shp extension, configurable!.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Services</title>
<style type="text/css">
	body { font-family:Verdana, Arial, Helvetica, sans-serif; }
	h1 { color:#7F983C; }
</style>
</head>
<body>
	<h1>Service</h1>
    <p>
      <a href="index.htm">Home</a> |
      <a href="public/default.htm">Public</a> |
      <a href="members/index.htm">Members</a> |
      <a href="about.htm">About</a> |
      <a href="service.shp?num1=1&num2=2">Spin Services</a> |
      <a href="login.htm">Login</a>
    </p>
	<!--[content]-->
</body>
</html>

When a .shp file comes down the pipe:
* The file name is mapped to a method.
* The method loads dynamic content into a buffer
* The requested file is loaded from the SD card into HUB memory
* The  placeholder is replaced with the dynamic content
* The file is written back to the SD card as temp.htm
* temp.htm is served up

I still have a lot of work to do but it's a start.

I have one known issue. for some reason IE and Firefox will not render the page. In Firefox, if I view the source I can see the html but the page itself is blank. I'm thinking it has something to do with the header but I'm not sure. Maybe available ports?

CassLan · 2010-12-11 05:51

Mike, I have been totally swamped and have not yet had a chance to work with your latest code, but I'm looking forward to it

RIck

Mike G · 2010-12-11 06:06

Cool let me know what bugs you find or suggestions you might have. I actually have updated the code just have not posted it. I'm thinking of putting the code on http://code.google.com/ or some other collaborative source control location.

$WMc% · 2010-12-11 13:29

Mike G wrote: »

The attached is a mixture of stuff from Roy to Tim. I added simple parsing of the HTTP header so we can grab HTTP GET or POST name value pairs.

request.__GET(string("name"))

Still a little raw but it works. The __POST method needs some work... It has to do with finding the header body, 0x13, 0x13. [Fixed:] I was trying to match the pattern 0x13, 0x13 to find the request body. Changed the pattern to 0x13, 0x10, 0x13, 0x10 and that works great.

Updated (11/23/2010) a object a bit and cleaned up the code. Make sure you set your IP, MAC etc. I made a host file entry to resolve "spinneret", so make sure you edit the HTTP FORM node according to your setup.

This object parses the HTTP header and buffers the request URL, just in case you need to know the last page served. Since I'll be using XMP RPC, I determine the file type. If the type is "spin" then I'll run the appropriate service method.

The MatchPattern method sets up starting and ending memory pointers. That part might be a little confusing when you're looking at the code.

[11/26/2010] spnneret.zip contains HTML pages

[11/27/2010] Added a simple dispatch method to load dynamic content
[11/28/2010] Added dynamic file generation

'
Hello Mike
'
I'm following your work with the Spinneret.
I'm getting an error on line 372 of the HTTPdemoMG spin file.I tried to fix-it but nothing I do seems to work.
'
Here's a screen shot of the Error.
'
Thanks for any help.
'
Great Work btw!

Mike G · 2010-12-11 17:48

You're probably missing or need to modify the following method in SD2.0_FATWrapper.spin. I modified the file that Roy created so I could open a file for (r)ead, (w)rite, or (a)ppend.

PUB openFile(fileName, action)
  return fat.openFile(fileName, action)

I took a look at the zip file and it looks right but I might have messed something up when I zipped the project.

$WMc% · 2010-12-11 20:45

Mike G wrote: »
You're probably missing or need to modify the following method in SD2.0_FATWrapper.spin. I modified the file that Roy created so I could open a file for (r)ead, (w)rite, or (a)ppend.
PUB openFile(fileName, action)
return fat.openFile(fileName, action)
I took a look at the zip file and it looks right but I might have messed something up when I zipped the project.

'
MikeG:
Thanks for the reply.
'
That fixed the error at line 372,Thanks
'
Know I have an error at line 412
412: SDCard.writeData(@data, strsize(@data))
'
The error I get is "Expected a subroutine name"
"writeData" is the one in question.This is in the HTTPDemoMG.spin program
'
I've never used an SDcard before and this is were the errors seem to be.
'
I don't have any SD card hardware other than whats on the Spinneret. So I can't trouble shoot any code easily with the Prop.
'
If you could help Me out a little more, I would really appreciate it.

Mike G · 2010-12-12 07:33

I downloaded and unzipped the project file contained in Spinneret-Get-Post-Demo.zip. Then opened "HTTPDemoMG.spin" and pressed F9. It worked...

I imagine you're not using the SD2.0_FATWrapper.spin file provided in the zipped project. The error means, "Hey, where the heck is SDCard.writeData"?

The demo requires an SD card loaded with the sample web site. If you did not purchase a micro SD card, copy the spinneret web site to the card, and insert the card into the slot on the Spinneret, the demo will not work.

$WMc% · 2010-12-12 10:04

Mike G:

I missed the DS1307_RTCEngine.spin OBJ. in the SD2.0_FATEngine. I don't have it in my Library and I didn't see it in the OBEX. I'll have to dig around for it.
'
Thanks

Mike G · 2010-12-12 10:23

I added my lastest stuff to this post.

Everything you need is in the zipped file in this post. Simply download, unzip, update your ip configurations, and load. You should not have needed the DS1307_RTCEngine.spin file as that was commented out of the SD2.0_FATEngine. Looks like you're mixing and matching files.

$WMc% · 2010-12-12 11:24

Mike G
'
You were right about the RTC being rem out.I now have my glasses on.
I rechecked the SD card, I looked a lot closer this time. I found that when I sent the files to the SD card the I.E. files got converted to google chrome files.I don't know how google managed to do this,but it was a good one.
'
I'll get this figured out
'
Thanks for posting the whole project and Thanks for the replies.

Mike G · 2010-12-12 11:33

The files in this post have an htm extension. Sounds like Chrome is your default htm file viewer?

$WMc% · 2010-12-12 13:21

Mike G wrote: »

The files in this post have an htm extension. Sounds like Chrome is your default htm file viewer?

'
Yep! my son likes chrome for some reason, and he uses my laptop.
'
I got the files rite on the SD card, still wouldn't compile HTTPDemoMG. I deleted all the files and reloaded what you posted. Now every thing compiles good.
Thanks
'
You are the Man !

Mike G · 2010-12-12 14:39

So you see that HTTPDemoMG is no longer in the project? It has been replaced by HTTPServer and HTTPClient, right?

$WMc% · 2010-12-12 16:49

Mike G wrote: »

So you see that HTTPDemoMG is no longer in the project? It has been replaced by HTTPServer and HTTPClient, right?

'
Yes, but it took me a minute to see what you had done since it was still in the new Zip.
'
I see why you got away from port 80. Using this port will shut-down a lot of app.s on windows. I changed the port forwarding to the 5000 range like you suggested.Know my LAN doesn't crash, Thanks for this heads-ups.
'
I have one question about " destIP[0]- [3] " and 184.106.153.149. What is this IP address? (HTTPClinet lines 244-247)
'
Thanks for the replies and the help!

Mike G · 2010-12-12 18:32

The IP address goes to thingspeak.com. Someone else was asking how to use the spinneret as a client and they wanted to hit thingspeak.com.

Simply put http:// in front of the IP address in your browser to see where it goes.

I didn't get away from using port 80, my ISP blocks port 80 traffic. So if I want to hit my spinneret from outside the network, I have to forward the traffic. Same goes for using the spinneret as a client. Set it to some unused port other than 80. That way the response makes its way back.

Mike G · 2010-12-19 07:59

Made a simple update to the HTTPRequest object and web application. I did not realize that IE would always look for an ico file. So, if you don't have an ico file in the application root and you don't implement a file not found... well you're asking for trouble.

Mike G · 2010-12-23 08:18

Added support for IsPostBack. I'll leave the server up for a few hours this morning. Play around... by the way, I still need to implement encoding and decoding.

URL: http://spinneret.servebeer.com/index.htm or http://spinneret.servebeer.com:5000

I'm curious to see what happens.

Mike G · 2010-12-23 12:50

Well, the folks over on the the Propeller forum stress tested the Spinneret. I'm happy with the results using only one socket. Requesting small pages seemed to work out fine. Downloading the Spinneret manual (500K) caused some lockups. Not to mention I had a bad link on one of the pages.

I have a lot of work to do, validation and encoding/decoding are probably next on the list. Then, I'm looking at using multiple sockets, saving configuration in EEPROM, XML remote procedure call (RPC) with XML style-sheet transforms (XSLT), and a little AJAX. After that's all done, I can start my project

Mike G · 2011-01-01 15:50

I added URL decoding (Roy's code) and page not found.

Roy, if you happen to see this, I made a small change to your DecodeString method when dealing with %##. The "inPlace++" line after "if (char == "%")" caused a little sync issue.

PRI DecodeString(decodeStr) | char, inPlace, outPlace
  inPlace := outPlace := 0
  repeat 
    char := byte[decodeStr][inPlace++]
    if (char == "%") ' convert %## back into a character
      'inPlace++ ' skip %
      ' first nibble
      char := byte[decodeStr][inPlace++] - 48
      if (char > 9)
        char -= 7
      char := char << 4
      byte[decodeStr][outPlace] := char
      ' second nibble
      char := byte[decodeStr][inPlace++] - 48
      if (char > 9)
        char -= 7
      byte[decodeStr][outPlace++] += char
      ' since we trashed char doing the decode, we need this to keep the loop going
      char := "x"
    elseif (char == "+") ' convert + back to a space
      byte[decodeStr][outPlace++] := " "
    else ' no conversion needed, just set the character
      byte[decodeStr][outPlace++] := char
  until (char == 0)
    
  byte[decodeStr][outPlace-1] := 0 ' terminate the string at it's new shorter size

Mike G · 2011-01-01 15:58

Who's hitting the my Spinneret with only GET / HTTP/1.0? Just curious if it's a real person.

JAGO · 2011-03-03 08:37

Hi Mike

I'm follow your work with the spinneret and i think that it's awesome... and i have a cuestion about the real time clock, i want to integrate the date and time in the web page but i dont have much experience in spin program..can you give me some help?? or give me some orientation about this in your code??

Many thanks, mike..

Mike G · 2011-03-03 17:29

Sorry; I don't have the RTC wire up in the code. I plan to add the RTC in the next version.

or give me some orientation about this in your code

Anything specific?

JAGO · 2011-03-04 06:30

Hi mike,

Many thanks for your answer... I know that your code has not RTC, my idea is to integrate this function.. and with this text "or give me some orientation about this in your code" i meant to give me an idea of where or how insert this function in your code and in the web page...well i hope you can give me some help..

Many thanks... and greetings

Basic GET and POST Object [Updated 1/1/2011]

Comments