Shop OBEX P1 Docs P2 Docs Learn Events
How to download a webpage into RAM buffer with PropNIC? — Parallax Forums

How to download a webpage into RAM buffer with PropNIC?

MicrocontrolledMicrocontrolled Posts: 2,461
edited 2010-05-04 19:47 in Propeller 1
I have recently received my PropNIC and so as to not reinvent the wheel I was wondering if anyone had some sample code for simply downloading the web page (source) at a web address stored in a string? Then downloading straight into a RAM buffer for reading? The download can be all at once for it is not made for big webpages to be loaded. I am not knowledgeable of Ethernet works, so if you want to say some kind of AJAX or IP something or other, I may be clueless. smile.gif

Thanks in advance,
Microcontrolled

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Check out my new website!!

Use the Propeller icon!! Propeller.gif

Follow me on Twitter! Search "Microcontrolled"

Comments

  • MicrocontrolledMicrocontrolled Posts: 2,461
    edited 2010-05-04 18:13
    I really need some help here. I am using Harrison's TCP/IP stack program. Anyone know how to do this?

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    Check out my new website!!

    Use the Propeller icon!! Propeller.gif

    Follow me on Twitter! Search "Microcontrolled"
  • heaterheater Posts: 3,370
    edited 2010-05-04 18:48
    You might want to read "HTTP Made Really Easy" here: www.jmarshall.com/easy/http/

    It has a sample GET request and expected response about halfway down.

    I would suggest you start by experimenting with a manual telnet connection to port 80 on the server you want to read from. Do something like this on your PC and see what come back:

    GET /path/file.html HTTP/1.0
From: someuser@jmarshall.com
User-Agent: HTTPTool/1.0
[noparse][[/noparse] blank line here ]

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    For me, the past is not over yet.
  • MicrocontrolledMicrocontrolled Posts: 2,461
    edited 2010-05-04 19:00
    Thanks! So would I send that as a serial command using the telnet object?

    I typed the code into command line (DOS) for testing and it returned an error. What did I do wrong?

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    Check out my new website!!

    Use the Propeller icon!! Propeller.gif

    Follow me on Twitter! Search "Microcontrolled"
  • heaterheater Posts: 3,370
    edited 2010-05-04 19:03
    OK Here is a simple example that works for Linux Weekly News at lwn.net

    Open a TCP connection to lwn.net on port 80.

    Send the following lines:

    GET / HTTP/1.1                                               
Host: lwn.net



    Note the blank line at the end.

    LWN replies with:

    HTTP/1.1 200 OK
Date: Tue, 04 May 2010 18:55:44 GMT
Server: Apache                     
Expires: -1                        
Content-Length: 23787              
Connection: close                  
Content-Type: text/html; charset=iso-8859-1



    Followed by the web pages HTML content.

    As I say it helps to experiment with this using telnet from a DOS box or from Linux first.
    Replace "Replace "GET /" with "GET /whatever/page/you/want.html" for your own pages.

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    For me, the past is not over yet.
  • heaterheater Posts: 3,370
    edited 2010-05-04 19:07
    Sorry when I say open a telnet connection I mean use the "telnet" command in your DOS box. I have no Windows or DOS here but under Linux I would type:

    telnet lwn.net 80

    at the command prompt and then when telnet says the connection is made type the "GET...etc..etc"

    You might want to check how to use telnet under DOS I'm sure it's a little different way to give the port parameter.

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    For me, the past is not over yet.
  • MicrocontrolledMicrocontrolled Posts: 2,461
    edited 2010-05-04 19:17
    This makes more sense. So I can send the same lines using the Propeller's telnet object and get the same results?

    Thanks,
    Micro

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    Check out my new website!!

    Use the Propeller icon!! Propeller.gif

    Follow me on Twitter! Search "Microcontrolled"
  • heaterheater Posts: 3,370
    edited 2010-05-04 19:23
    That's what I'm hoping. Did you get it to work from telnet?

    I have no idea about the telnet object or PropNIC but I'm hoping it has a method to "open" a TCP connection to a specified server, probably using the IP address rather than the servers host name. There should be a way to specify the port number to connect to (80). Then methods to "send" some strings or lines and receive whatever comes back.

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    For me, the past is not over yet.
  • MicrocontrolledMicrocontrolled Posts: 2,461
    edited 2010-05-04 19:33
    How would you get the IP address? Or for that matter the server name?

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    Check out my new website!!

    Use the Propeller icon!! Propeller.gif

    Follow me on Twitter! Search "Microcontrolled"
  • heaterheater Posts: 3,370
    edited 2010-05-04 19:47
    Well if you want to visit Parallax you might type "http://www.parallax.com/" into your browser.
    So "www.parallax.com" is the server name we are looking to connect to and the "http://" specifies the HTTP protocol which is generally operated on port 80.

    Now to make the connection the browser has to find out the IP address of www.parallax.com, generally by asking a domain name server (DNS). The IP address happens to be "67.104.29.61".

    Manually we can get the IP address by using the ping command from the DOS box or linux command line. For example try:

    ping www.parallax.com

    and it will reply:

    PING www.parallax.com (67.104.29.61) 56(84) bytes of data.

    Giving us the IP address we want in brackets.

    Now often if a system allows you to use an "open" method with a server name string and a port number , say:

    open ("www.parallax.com", 80)

    one can replace the name with an IP address in the string like so:

    open ("67.104.29.61", 80);

    I hope your PropNic has this same idea.

    ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    For me, the past is not over yet.
Sign In or Register to comment.