Spinneret as a client?
homosapien
Posts: 147
Is it possible to use the Spinneret to parse specific data from an (externally) requested webpage? I have in the past used the Spinneret a server, but I don't think I have ever used it to navigate to a website and request a document (which, I guess, would be being a client...). I know this is done in a fashion for the SNTP server, but I am pretty sure the data returned from NIST webpage is just a tiny time string.
I am trying to query my local router with a Spinneret and get the current IP (global) address for my local network. I can enter the local address for the router in my browser (Chrome) and get a http page that shows me the stats of the router, included in which is the current IP (global) address.
However, if I look at the http source code for the returned page, it is pretty massive (1000's of characters), so I think the Spinneret would have to do some on-the-fly parsing to only retain the desired IP address and not all the other undesired data (which possibly would overflow whatever buffer the Spinneret is using?)
Is it possible to parse returned http data on the fly with the Spinneret, or is the returned data stored somewhere in a limited size buffer, which would preclude parsing on the fly?
Thanks,
Nate
I am trying to query my local router with a Spinneret and get the current IP (global) address for my local network. I can enter the local address for the router in my browser (Chrome) and get a http page that shows me the stats of the router, included in which is the current IP (global) address.
However, if I look at the http source code for the returned page, it is pretty massive (1000's of characters), so I think the Spinneret would have to do some on-the-fly parsing to only retain the desired IP address and not all the other undesired data (which possibly would overflow whatever buffer the Spinneret is using?)
Is it possible to parse returned http data on the fly with the Spinneret, or is the returned data stored somewhere in a limited size buffer, which would preclude parsing on the fly?
Thanks,
Nate
Comments
If you are looking for your ISP assigned IP, I usually create a simple page that returns just the IP. Feel free to use this one...http://www.agaverobotics.com/spinneret/myip.aspx
Finding text on a page is straight forward. Retrieve a block of buffered data (HTML), Look for the text, get the next block of text, look for the text. Keep in mind the search text might exist on a buffer boundary. You'll need to keep some bytes at the end of the buffer so you have a little overlap.
Why do you need the WAN IP? What are you trying to do?
I looked on the Google code page, was a little overwhelmed with the number of objects and I was not sure what each one does. However with a little searching I found the 'Ethernet Orb' project and discovered it is built on your TcpSocketClienDemo.spin and think I have idea how to approach the task now.
As to what I am trying to do: I have a Spinneret collecting data which is accessible on the local network (via mobile device/Android phone), but I wanted to be able to check the data remotely (ie global internet). However, the local network has a dynamically assigned IP address (it changes every so often) which is a problem. I was working on a way for the Spinneret to detect the IP address of the local network, and when it changes send an email to the remote device. This way when the remote device wants to connect, it will simply check for the latest email/IPaddress from the Spinneret and connect using that.
I realize that this topic has been broached before, and the accepted method is to have the Spinneret upload data to a fixed web database, and then remotely query the database. However, I don't (yet) know how to do this, so I thought I would try my idea before getting into even more new material (establishing a web database, maintaining it, uploading to it, downloading from it etc.). I am hoping the 'send the latest IP address' method will take one link of complexity out of the equation (or maybe it will make it even more convoluted, that seems to be my SOP...haha)
How do you determine your own IP address with your myip.aspx page? I was planning on simply querying my local router, which does have a page that shows the IP address as seen from the internet, and extracting that small piece of data from there. Is there a better way?
Thanks,
Nate
I use a free Host/Redirect service from NoIP. Basically, you can select a host name and redirect to any IP. NoIP has a free client app that runs on a PC which syncs your WAN IP to the host name. Similar to what you're trying to do without the email.
Very doable and a lot of moving parts. It's a great educational experiment and I went down this path a few years ago. I learned NoIP was a lot easier and several less things to worry about (troubleshoot).
I have a different perspective because I've been down the path, tried a different entrance, and banged my head against a tree... IMHO, your approach is complex
No magic - one line of code unless your behind a proxy then it's 5 or 6. All web server languages are pretty much the same, just different syntax.
The source IP address is part of all client requests; otherwise the server would not know where to sent the response.
I have observed that my ISP-assigned LAN IP address changes every day. The first step in this project will be to monitor it and see exactly when/how often it is changed (and yes, in the back of my head is the thought that simply by 'pinging' for my address, I may actually cause the address not to be changed. I don't at present know what usage criteria the ISP uses to decide when to not re-issue a lease).
EDIT - I just had a 'Duh' moment, just realized your 'myip.aspx' page returns MY IP address, which was parsed from MY initial page request. I was thinking it was somewhat magical because it was determining YOUR IP address from behind your router...
Believe or not publishing my WAN address is just as simple.
Have at it and let us know how it goes... learning is the most important thing.
Nate
No, the underlying SPI drivers are specific to the Wiznet chip. The protocols are different - see the spec sheets.
You probably read that the higher level libraries like DHCP can be use on either the Spinneret or W5200 for QuickStart by using the appropriate driver W5100 or W5200. Sorry this was not clear.
Last time I used the Spinneret, I was using the httpServer code, these updated objects are great! The distinction between objects/protocol layers make for understanding the stack much easier.
Nate
I have a program that runs successfully as a server, it prepares and opens a socket that waits, listening for client webpage requests.
I have another program that runs as successfully as a client, it periodically (once per minute) sends a request for a page from the LAN router, parses the LAN IP from that received page, and stores it in Propeller memory (basis of webpage sent from Server).
I'm thinking about melding the two programs to run as one on the Spinneret. The simplest thing would be to have the Server program do its thing, and then periodically stop listening for webpage requests, become a client and get the current IP from the LAN router. Seems doable. Only problem is if a page request comes in while the program is in the client/parse mode. This probably is not a huge issue, the client mode would only be one second or so out of every minute, but this may be more of an issue when I want to add more functionality to the program (such as emailing the updated IP address) which might take more time from the webserver mode.
I'm wondering if there would be a way for the program to have the two modes (client/server) to run simultaneously in two different cogs. I believe the WS5100 has four sockets, not a problem there. However, there is only one SPI bus, and there might be a problem if the two cogs tied using it at the same time. I'm thinking there needs to be software control of the SPI bus (ie a lock) to prevent this, or is this not a problem? Is there already some sort of SPI control built in?
Nate
Sure, you can lock the SPI bus at the driver level or at a processes level. The SPIN manual has great info on using semaphores.
There's also simple old top down code where the a single process ping pongs between request or response mode. You could create one process that moves data between the W5100 and memory and another that processes the data. You can look at how the web server handles DHCP and SNTP during runtime.
How many current requests are expected? What's being requesting? Is the request coming from a browser? Have you tried testing this scenario?
I do not anticipate a lot of traffic on this project, but was just thinking ahead and trying to keep the code somewhat modular for other projects that might have larger needs. That said, I decided to go for the 'one socket, many different modes' method at this point in time to keep it simple for my linear-thinking mind.
Thanks to your help, I have made a ton of progress. The Spinneret in Client mode queries the LAN router, and by parsing the 'Connection Summary' page is able to determine the global LAN IP address successfully. This is put up on a internally generated webpage, which the Spinneret serves up upon request. So far I have not had any issues with page requests not being handled because the Spinneret is busy determining the LAN IP, probably because it is only in the Client mode (to get the Connection Summary page and parse it) for about 1 second out of every minute.
I have also successfully implemented emailing into the program. I read through some of the threads on the forum and did some external research on SMTP (best quote I found - "Nobody ever sticks to the standards, it's a zoo out there") and was anticipating some hair-pulling to get it up and running, but it went practically without any hiccups - the program I am building also now sends an email to my gmail account (android phone account) with the latest LAN IP address if there is a change (also sends a 'heartbeat' email once per day).
Of course, now that I have the Spinneret running 24/7, the LAN IP address has stopped being changed every 24 hours...but I am OK with that, past experience has shown me that you can never make a deployment too bulletproof. I will post the working code after I clean it up a bit, it is the software equivalent of my workstation after a string of all-nighters before a deadline...Will probably be into next week before I get a chance to work on the Android app that will parse received emails to determine latest 'good' website address.
Nate