Chunking Data transfer
oodes
Posts: 131
Hello Guys ,
I dont have the exact source code on me its in the office at work but I was having some issues today when trying to chunk data across from Spinneret to another Spinneret. I 'm using one as the Http server. In the ClentGet on the Serve Beer website I'm using the 'GetMyIp' function to contact another spinneret and transfer data. The transfer works fine when the packet is smaller than the rxdata buffer. When the data is larger Im trying to send it in 1k byte chunks cant seem to get it working
I dont have the exact source code on me its in the office at work but I was having some issues today when trying to chunk data across from Spinneret to another Spinneret. I 'm using one as the Http server. In the ClentGet on the Serve Beer website I'm using the 'GetMyIp' function to contact another spinneret and transfer data. The transfer works fine when the packet is smaller than the rxdata buffer. When the data is larger Im trying to send it in 1k byte chunks cant seem to get it working
bytefill(@tempBuff, 0, TEMP_BUFFER) ' Wait for the response repeat until size := Socket.rxTCP(id, @rxdata) { Here the slave spinneret is sending the data after the connection is established. Obviously the first 1k chunk is getting across and I'm logging it the logfile.} bytemove(@tempBuff, @rxdata, strsize(@rxdata)) pst.str(@tempBuff) AppendLog(@logfile)I need to read in 1k of data into the @rxdata , log it in the AppendLog bytefill the buffer with 0's and keep repeating this until all the data is transferred. I know this is a trivial matter but I cant seem to get it working
Comments
I have to ask, why transfer data from one Spinneret to another? Does the second Spinneret process the data?
Using one spinneret as a expander board to my existed prop devices as all the I/O's are in use , this also enables me to use an Ethernet connection to the device. The httpserver is running a website and requests and view the logs from all the networked prop devices.
I don't know the project requirements and I don't want to tell you how to design a solution but why send logs from one Spinneret to another? I see the multiple Spinneret as an ideal situation because the logs are unique to the Spinneret. Sending files to a centralized Spinneret means extra code to deal with the batch job and logic to identify the file source. The logs are always available via HTTP GET and a browser.
I would stay away from a batch job and send log records via HTTP GET or POST in real-time to whatever server. The out-of-the-box HttpServer can easily handle one record at a time and up to 1k as you have identified in post 1.
Thats a great idea. Thanks
I all for advice Mike and design ideas, you may have guessed but I a bit of novice :P . Do you mean that by having the Http Server running on one spinneret that I can view the logs stored on the sdcards of the networked spinnerets using the HTTP GET and not having to transfer it ?
Ah you mean send the log entries to the server as they happen instead of saving them on the sdcard of the 'slaves' and trying to chunk the data like im doing? And then I can view the Logs on the server?
Any Spinneret running HttpServer and storing logs on an SD Card is accessible by a browser. Moving the files to another Spinneret is extra work unless the target Spinneret is processing the files in some way. Again, I don't know the project specifics but it seems like a lot of extra work for no reason. Is there a reason to batch the log files?
Quick question. Can you tell me why that when I access a file on the Spinneret from my browser sometimes it downloads the file and other times it just is viewed on the window. Is this to do with the html code I'm using or the spinneret. Seems to do one or the other at different times if I change bits of code in my source spin file.
Is the behavior the same in all browsers?
What file content causes the file to download?
What file content causes the file to display in the browser window?
HttpServer looks at the file extension to send the appropriate content-type header. Is the file extension always txt?
What behavior do you want; download or view?
On the sd card are some text file name as in the Picture.
http://93.107.65.84:5000 and enter the password 'desmond' . If you click on the Logfiles tab at the top . If you click on PROGEN1 (LOGFILE1.TXT) or PROGEN2(LOGFILE2.TXT) on the Left hand side in the logfiles page you will view the .txt files in the browser. If you click on 'VIEW THE LOGFILES' (LOGTEMPS.TXT) you will download the .txt file.
Don't mind my crappy webpages .
I want to download the file , not view it
I normally run Chrome but I just checked it with Explorer and its the same outcome and files are always .TXT . Content is always lines of text unless some unprintable characters make there way in there which can happen. I originally thought it was to do with the size of the file but this is not the case
DEs
Or right click and save.
Back again. I've migrated my code from the Http Server to the new WebServer and I have a quick question.
On the old server the Staticfilehandler was called at the end of the Dispatcher(if not returned first). On the new Server RenderFile now does the job of the StaticFilehandler and RenderDynamic handles anything that isnt static. So on the old server from your examples on the ServerBeer it is possible to intercept the RequestLine for the example of turning on and off a LED.
My question is that I want to parse the querysting from a static file? Like if I have in the html code The render file will always return the led.htm and I want to get to the querystring
Does my Question make any sense?
It is located in the HttpHeader in the TokenizeHeader function? How might I extract this and use it?
Des
I was making the call from the wrong place.
Think this is right, working anyway
Thanks
Do you see the four example filters in RenderDynamic?
maybe.
SO I have a webpage called delete.htm The static file delete.htm is returned by the Renderfile which is fine for the initial display . But when I am attempting to get to the querystring on the last line of the html , the Renderfile returns delete.htm due to the If else statement. You will to excuse my ignorance on this Mike.
i.e. You could create a filter that looks for that returns an XML response to the caller.
I need to look more into the XML end of things and the web side based programming. ROLL on the christmas holidays.
Thanks again
Another unrelated Question I came across recently maybe you could shed some light on. If I have a buffer located in the DAT section, say a 1000 bytes and at some point in the Program I do a bytefill of a 1000 0's to clear the buffer , i notice that sometimes this will clear what is located in the DAT section directly below it. Took me a long time to figure out what was happening. e.g below: If I bytefill Logdata with zero's it was also clearing what was in the ENQ location. I noticed this happening also with some elements I had in my VAR section also.
One of my spinners also sometimes draws an extra 100mA of current after powerOn , program in the eeprom runs but ethernet connections are unresponsive . You ever come across this?
What's the spinneret doing on start up? DHCP, SNTP, initializing hardware???
This is a modified version of GetMyIp (Modifed probably not in a good way but it works ), don't waste much time sifting to my rubbish code
Pretty much all of the above. Uses your INIT function from the webserver, Starts 2 more cogs , one using FULLDuplexSerial for Comms with another device and another for handling logEntries.
This just takes whatever is in the Rx buffer and plops it into LogData. There no "Size" limit checks.
Do ya mind me asking , do you work for Parallax or are ya just really into Parallax devices?
Still running into this problem with the extra current draw. I'm thinking its the Spinneret , all cogs are starting fine i have debugged them 7 in total(4 original from webserver , extra SDcard(5) , comms with another Prop using FullduplexSerial(6) and a LogfileHandler(7) but it will hang at
when the extra current draw is occurring. This only seems to happen after long stages in Power Down (Overnight) . Could it be related to the charging of the super Capacitor? I also had other code running on this same spinneret a few months ago and it was starting ok but this sporadic probem would occur and could not connect to it when all it was doing was waiting for a connection. Problem only occurs at Startup and seems to be Ethernet related so if it is hardware must be either the Capacitor or the Ethernet PHY
I have loaded exact same software (bar the MAC address) onto a different Spinneret and its was working fine and then when I loaded back up the original one that was causing problems starting working again. Its a really sporadic problem.
I remember in one of your posts you mentioned blowing a Spinneret , what was the symptoms do you recall ? I'm just interested to know if you ever had any similar problems and whether I need to bin this spinneret
Des
Anyway, you have the error trapped. Retry the the DHCP process when you get a discovery error. Since the issue is intermittent, it's probably a timeout. Timeouts are configurable in the socket object: TRANS_TIMEOUT.
, I feel bad putting up code and expect someone else to sieve through it.(I wont hesitate next time) I was just interested to see if anything similiar had happened.
When the extra current draw is happening I can retry it a 100 times and it makes no difference. I have to start disconnecting Ethernet cable , remove the power cables , restart and eventually it works , might be 1 min might be 3 hrs.
Source to fix the record
I can email it to you if you like