HTTP DNS client with prop and 5100

R Pankau

I'm looking to build a project with a Prop and a Wiznet 5100 via SPI. The ultimate goal is to query a website for a web page that can be parsed so as to find a few nuggets of numeric data. I've done this with VB.NET and Python but I'm finding that doing the same thing with the 5100 is quite a different animal. which is ok, I just need to start eating the elephant at a logical place.
So taking the first baby steps I believe what needs to be done on a high level is this:
1. setup 5100 with mac, and ip addresses etc.. (there are some good examples that I've found)
2. Establish a socket with DNS server (also good examples available)
3. perform query, handshaking, etc... with server hopefully endup with IP address of target website. (need some details here)
4. Establish socket with target website
5. request web page. (send url string at this point???)
6. parse
7. rinse repeat.

Can anyone steer me in the direction of some byte for byte examples of the above paragraph? web sites or books, I have Fred Eady's book "Networking and Internetworking for micros" this is a good book but it does not really explain how to roll your own dns client and http client. I know that these topics are huge but surely it can be condensed to perform the simple tasks that are required.

thanks in advance.
Randy

Nick McClick

This doesn't answer your question, but;

Are you committed to the Wiznet? The ENC28J60 has more mature propeller libraries & there are several code examples (Harrison's PropTCP Telnet Server, ybox 2) doing exactly what you're looking for.

R Pankau

I took a look at the ybox and it was fairly impressive. The 5100 is not necessarily the network adapter of choice for me, although I did already purchase one. The ENC28J60 is not overly pricey so I may switch.

thanks for the help!

w8an

I've used this W5100 Ethernet Driver.spin v.1.2.4 in a couple of projects and have found it works just fine. It doesn't appear to be in the Obex however.

Steve

Use the attachment by Patrick1ab posted 11-18-2009, 01:42 AM

R Pankau

I'm digging in a bit and using Wireshark packet sniffer to decode what a real HTTP client does. Can you recommend a good UDP/TCP generator tool? I've tried this one, UDP Test Tool 2.5 by Simple Com Tools which is listed in the comments section of the Brilldea test code but it is having trouble working on my machine, will not install.

R Pankau

So things are going well so far. I put together a DNS query to get the IP address of the web page I'm looking for. (Will post code later tonight). I tried it and received an error response from the Domain name server, which was actually encouraging. I'll post the string that was sent as well as the string received.

Has anyone done this with a 5100? I did see the example with the Microchip part (Ybox) but I can't tell yet if the error is in the payload or something with the Headers in the 5100.

smbaker

R Pankau wrote: »

I'm digging in a bit and using Wireshark packet sniffer to decode what a real HTTP client does. Can you recommend a good UDP/TCP generator tool? I've tried this one, UDP Test Tool 2.5 by Simple Com Tools which is listed in the comments section of the Brilldea test code but it is having trouble working on my machine, will not install.

HTTP is generally pretty easy to code once you realize the subset of it that you usually need to use. For example, telnet to www.google.com, port 80, and type the following with a blank line after it.

GET /index.html HTTP/1.1
Host: www.google.com

The first line will be a status line and usually looks something like this for a successful reply:

HTTP/1.1 200 OK

This will work for many servers, although some will require additional headers. In particular you will occasionally need an 'Accept */*' or a 'User-Agent: ' (snoop a connection for a valid user agent string). Some web servers may be configured with referrer checking on important pages or especially on images, which can be really irritating.

One issue you may have is with page length. Last time I used the ENC28J160 driver it did not do TCP fragmentation, and this put a limit on how large of a TCP message you could handle. I'm not sure if the state of the stack has changed since then.

R Pankau

thanks for the tips smbaker.

DNS query is attached.
Also a screen shot showing the UDP payload of the DNS response. the ID was $1234 in bytes 8 and 9, then the next two bytes are 81 01 or 10000001 00000001 meaning that it is indeed a response (first bit) but that last bit of the second octet indicates a "Format Error" The name server was unable to interpret the query.

If I dump the query payload to the terminal it looks good, exactly like the wireshark capture.

R Pankau

Hey It works!
I had the payload length too short by one byte in bytes [6] and [7] of the data.

R Pankau

So I was able to get the DNS to work and now I'm working diligently on a GET request.

The TCP is confirmed Established before sending this and to check for a response I'm polling _S0_RX_RSR0 for the length of a response greater than zero and nothing so far.

The user Agent line and most of the rest is stolen from a Wire-shark grab of what my browser generated.


GET / HTTP/1.1

Host: www.powersmartpricing.org/chart/?display=table

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 115

Connection: keep-alive

R Pankau

GET request works but a bit unusual. the 5100 "should be" set to 8K on the receive buffer size. Set RMSR to 0xFF which should be 8K receive and only port 0 will work.

The received size register always contains 2904 (if the data is 2904 or greater) and the rest of the web page appears to be truncated.

Is there a Command that goes with the HTTP GET request that can specify how the data is returned? ie no fragmentation since the 5100 does not support that.

I know I'm kind of off base now in the prop forum but there seemed to be a few 5100 users here who had good results.

Bob Lawrence (VE1RLL)

The HYDRA EtherX Card uses the same 5100 chip.

You may get a few ideas from the manual and code on the support website:

http://www.averydigital.com/products.html

Bob Lawrence (VE1RLL)

Propeller Web Server Software....

http://forums.parallax.com/showthread.php?p=892553

R Pankau

Thanks Bob, I'll take a closer look.
Seems like there are lots of examples of web servers with the 5100 but not much in the way of making it a client.