[Contest Entry] Real-Time Energy Price HVAC Controller
R Pankau
Posts: 127
My plan is to use the 5100 as an HTTP client to get real-time electrical rates from my electric supplier. With this data I can allow the spinneret to raise or lower the temperature setpoint in my home. The threshold values (temperature vs prices) hopefully will come from a web page served by the 5100 on a form.
As a bonus the spinneret could control multiple relays for controlling things like a dehumidifier for instance. No need to run it during the day when electricity is expensive, it could do all of the dehumidifying necessary in the wee hours of the morning when a kwh is usually less than $0.02
As a bonus the spinneret could control multiple relays for controlling things like a dehumidifier for instance. No need to run it during the day when electricity is expensive, it could do all of the dehumidifying necessary in the wee hours of the morning when a kwh is usually less than $0.02
Comments
My most recent brick wall was in the form of not being able to retrieve more than 8K of web page after issuing an HTTP GET request to a web server. As luck would have it the data I really need is somewhere past where the 5100's memory ends.
It would be really great if there were a way to retrieve the remaining portion of the web page with a subsequent GET request. When I see this on page 5 of the 5100 datasheet it worries me. "Not support IP Fragmentation" I assume that is a show stopper.
The HTTP syntax is relatively new to me so this may be a dumb question. Can I somehow issue the GET specifying where to start sending data? ie someplace not zero but more like byte 7,500 of the web page.
I've toyed with the idea of renting server space (probably can be had for free) and writing some python maybe to get the small amount of data that I really need and make it available via another very small web page that the spinneret could then query in full. This is new territory to me also.
thanks.
Have you checked if they have a 'minimal' interface for smartphones and the likes?
Possibly even some sort of IRC system?
Does not matter, same mechanisms. IP fragmentation is something completely different and got nothing to do with this. To avoid receive buffer overflow, TCP uses window size, forcing the sender to pause transmission until the receiver has processed its input queue. All this is handled by the TCP stack itself.
I've checked a little into a smaller page for phones etc.. but have not really found anything. Our System operator MISO, Midwest Independent System Operator does this very well with a small xml page but I believe it is only for the commercial and industrial producers or large users. I have not been able to find the consumer price on that page.
Here is where my consumer price is displayed
http://www.powersmartpricing.org/
Vs the place for showing the prices for each node at the source.
http://www.midwestiso.org/page/LMP+Contour+Map+%28EOR%29
https://www2.ameren.com/RetailEnergy/rtpDownload.aspx?Company=19&Date=12/05/2010&ptype=D
Not sure how this works. Seems like I thought about this once but the Download button is part of a POST command I'm guessing. So If I have to post back the entire page I'm out of luck because it's 56K in size.
I wonder if I can just request the CSV page directly without all of the web pages? Is that a normal function of a web server? If I know the name of a file like this csv in which the name has the date embedded, then should I be able to request it from the server? Is that a GET request?
What are the odds that I'm going to be able to request the csv with the 5100?
Simple, you do not have any 8kb receive limit at all. :-)
The 8kb buffer, means that after the sender has given you 8kb of data, you have to remove it from the 5100 before the sender delivers you the rest of the data you ordered. The 5100 continuously tells the sender the amount of space left in its receive buffer (via "TCP window size"), and you do not have to bother with this unless you are implementing your own TCP stack.
The 8kb buffer size will only limit the performance, not the size of the data your are able to handle. In theory, you could send TCP data byte by byte, but it would be terrible inefficient, but still able to deliver gigabytes.
You are describing a non existent problem. If you do not get it to work, there is something else you have overlooked.
Thanks, I'll let you know what happens...
I created a second buffer in the prop for testing thinking that maybe it was taking too long to clear the contents of the first go-round before calling rxTCP (rxdata2 here) but I get to the same stopping point after the first rxTCP call and then never get more data after that.
sock 0 status starts out as 0x17 in the repeat loop and then it eventually times out to 0x1C without giving me any more of the web page.
ETHERNET.rxTCP(0, @rxdata) 'copy 5100 received data to prop
PST.str(string("First Data in Buffer"))
PST.Str(@rxdata)
repeat
ETHERNET.readSPI(ETHERNET#_S0_SR, @sock_0_status, 1)
PST.str(string("Socket Zero Status Reg = "))
PST.hex(sock_0_status,2)
ETHERNET.rxTCP(0, @rxdata2) 'copy 5100 received data to prop 2nd time
PST.Str(string(PST#NL))
PST.str(string("Second Data in Buffer"))
PST.Str(@rxdata2)
bytefill (@rxdata2, 0, _bytebuffersize) 'clear prop buffer
while (sock_0_status <> $1D) or (sock_0_status <> $1C)
Known problems:
*When using UDP, there are buffer problems on packets over 1472 bytes. Under investigation. This problem may also exist when using TCP.
The data sheet is poorly written if you ask me, for the 5100 that is. I've been heavily relying upon the supplied driver because the 5100 datasheet was not user friendly. Maybe Timothy D. Swieter found this to be true also.
Generating the proper address for received data is a real circus , I am looking at page 51 of the manual "receiving process" Maybe I should try to understand this.
So when you send data over Ethernet using IP protocols and it's larger than the MTU, then it will be broken up into smaller chunks. Your receiving code needs to put the payloads back together to get the full data.
If you use TCP, then the packets you send/receive are guarranteed to be delivered in order. This is the easiest method for transmitting data between clients and servers.
If you use UDP, then you may get packets out of order, and you may not even get all of the packets. It's up to you to put something into the packets to indicate order, and to detect when a packet is not received and resend (usually done with ack packets). It's better to just use TCP for things like transfering files or large chunks of data. UDP is usually used for small packets of data that can tolerate being out of order and sometimes not actually make it through the wire.
Anyway, you are trying to use HTTP to get data, and it's likely that you just need to implement a little bit more of the HTTP protocol to get your stuff working. The headers can tell you more information about how the data is being transmitted. It may be in chunked format, which means you need to handle that. There are other things that can factor in also. Here is a link to a site that helped me get my HTTP demo working: http://www.jmarshall.com/easy/http/
Hope this helped.
With TCP handshaking etc... I would think that if packets were lost the server and 5100 would work it out by retransmitting.
The HTTP protocol has things like "keep alive" etc... that I'm not too familiar with.
Haven't ruled anything out yet.
I was going to tell you to look at the TCP Echo server code in OBEX about how you iterate rxTCP to receive the stream in whole, but when checking, I found the W5100 Indirect Driver 00.6 was quite buggy and didn't handle it well at all. So I fixed it.
Download the driver attached to this post, and look at the TCP Server Echo demo to see how to do it.
I was unable to actually use the fixed driver since my prototype is a prop proto board wire wrapped to a wiz811mj. I wrapped the SPI connection, it was much easier that way.
However I did use this
repeat until RX_size := ETHERNET.rxTCP(0, @rxdata[0])
ifnot ETHERNET.SocketTCPestablished(0)
quit
At first I found that the issue was that every time through the loop it appeared to grab the first part of the web page and spit it out again. So then I thought maybe I should not have increased the size of the receive buffer to 8K, One because it is not really necessary, and two the drivers assume the defaults.. so switching back to a normal 2K buffer (leaving _RMSR alone that is) it magically began working.
Just received 8814 bytes of success from a single web page.
Going into this fairly blind, the input buffer at 8K was actually a killer.
The driver in use is Brilldea_W5100_SPI_Driver_Ver005.spin, now I'm wondering if there is a bug in this that I should address?
Thanks for the help
Bug fixes:
- rxTCP return value may contain garbage.
- txTCP will deadlock if trying to send too much data at one and the same invocation.
R Pankau, keep in mind that TCP is stream oriented. No need fiddling around with buffer sizes and such, unless you know exactly what you are doing. But I guess you know that by now.
This also implies rxTCP may only return parts of what you are considering "all" your data. If you want 500 bytes from a web server, it is up to you to iterate rxTCP until you get all you need. There is no guarantee that you get everything at the first call, not even if the data is small enough to fit in one IP datagram. It is perfectly possible rxTCP only returns one byte at a time, however unlikely.
1. thinking that the 5100 receive buffer size needed to be as big as the total data to receive.
2. thus not realizing that repeated requests for more data would yield more data until the connection was closed by the server.
Maybe this not worth a large discussion but when I changed the buffer size to 8K either I did not do it properly or the rxTCP routine was not able to handle the increase in buffer size. as the data I was receiving appeared to be a lot of repeated data from the beginning of the HTTP stream. Now I see no good reason to change the buffer size of course.
Actually, you can not rely on a connection close neither. In HTTP, you have to check the "content-length" to know how much you are expecting, and other application protocols use similar mechanisms. HTTP/1.0 usually closes, but HTTP/1.1 may keep the connection open for reuse.
The ethernet driver may require some work before perfection, but when you start changing the hardware parameters, it is up to you to do that work.
This has nothing to do with the W5100, but with TCP. A TCP stream is infinite long, and you can never except to buffer it all, all the times. You always access TCP via a window into the stream.
Should I expect to see this "content length" in the response? As far as I know the TCP headers are hidden but what shows up visible to the application is the HTTP header and there does not appear to be any length.
The reason I ask is to know when to disconnect the socket. Would it suffice to keep checking for more data received, and if it returns zero for a specified time then disconnect? These are HTTP etiquette questions for which I should probably read up on.
Actually looking back on this thread, this was suggested and appears to be very helpful on the topic
http://www.jmarshall.com/easy/http/
maybe the Header, Content-Length:, is optional. I don't see it from my particular web site.
If your request results in a message body, there should be a "Content-length".
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 for HTTP/1.1 details.
For HTTP/1.0, if there is no content-length field, you can determine content length by the server closing the TCP session.
http://ftp.ics.uci.edu/pub/ietf/http/rfc1945.html#BodyLength
The header takes up a certain number of bytes. The header length is not calculated in the "content-length:" line but the header is part of the transaction. A transaction is complete when the server closes the socket after serving up the requested resource(s).