Lost Data in Transmission

oodes

Hello ,

I have a very random problem. I have a Spinneret running as a Server and 3 Spinnerets running as Clients logging data on the Server. Every 24 hrs I'm logging about 3000 entries in a text File on the Server , but over a 24hr period maybe 1 or 2 of those entries is missing some data bytes from the string sent. Sometimes this happens even less frequent , I had it running the last 72 hrs and it happened twice in around 10000 entries so its a needle in a haystack type problem.

First the code from the Client where the data is being sent:

repeat until not lockset(Collisionlock)
            case Collision
              0:
                lockclr(Collisionlock)  
                bytesSent :=Sock[0].Send(@tempProgenLog, strsize(@tempProgenLog))                     'Send the data to the host
                pst.str(string(CR,"No of bytes sent is: "))
                pst.dec(bytesSent)
                pst.str(string(CR,"Data Packet sent is..",CR))
                pst.str(FillTime(@time)) 
                pst.str(@tempProgenLog)  
                pst.char(CR)

The string @TempProgenLog is Null terminated I have added the logging entry shown in Cliententry.png confirming that the bytes are being sent.

On the Server side the code is as follows :

    repeat until sock[sockId].Connected
      sockId := ++sockId // SOCKETS
      if(++i//RTC_CHECK_DELAY == 0)
        rtc.readTime
        'pst.str(string("DHCP Time check: "))
        'pst.dec(rtc.clockHour)
        'pst.char(":")
        'pst.dec(rtc.clockMinute)
        'pst.char(CR)
        if(rtc.clockHour == dhcpRenew)
          RenewDhcpLease
          pst.str(string(CR, "Sync RTC with Time Server")) 
          pst.str(@divider)
          if(SyncSntpTime(SNTP_SOCK))
              PrintRemoteIp(SNTP_SOCK)
              pst.str(string("Web time.........."))
              DisplayHumanTime
          else
              pst.str(string(CR, "Sync failed"))
          Pause(delay)    
          ResetSntpSock(SNTP_SOCK)
          
        i~
    
    pst.str(string(CR, "Got  a connection"))
    'pst.dec(sockId)
    'pst.char(CR)
    repeat until bytesToRead := sock[sockId].Available
    pst.str(string(CR,"Bytes to Read is: " ))
    pst.dec(bytesToRead)
    'PrintAllStatuses
    
    'Check for a timeout error
    if(bytesToRead =< 0)
      pst.str(string(CR, "Timeout: "))
      pst.dec(bytesToRead) 
      PrintAllStatuses
      sock[sockId].Disconnect
      next 
       
    'Move the Rx buffer into HUB memory
    sock[sockId].Receive(@buff, bytesToRead)
    'bytemove(@TempBuff,@buff,6)
    
    'Display the request header  
    'pst.str(@buff)
    pst.str(string(CR,"Buff Data is:",CR)) 
    pst.str(FillTime(@time))
    pst.str(@buff)  

Here is where the bytes have been lost as can be seen from the logfile ServerLog.jpg

99.9% of the time the logs are fine but its this 1 random case where the bytes go missing. It will always log part of the string but some of it goes missing and its not always the same amount of bytes that goes missing.

Any thoughts?

Des

Mike G

Maybe, the receiver needs to wait until all bytes have been received.

oodes

Hello Mike,

Mike G wrote: »

Maybe, the receiver needs to wait until all bytes have been received.

Something like you have done here in the TcpSocketServerdemo?

repeat until NULL < bytesToRead := sock.Available 

Mike G

Not exactly. Generally, when sending data, the total number of bytes are send along with data. In HTTP it's the content-length header. This tells the receiver how many bytes to expect.

sock.Available reports the current number of bytes in the buffer. It does not know if all potential bytes have been received.

oodes

I'll have to have a look at that then so. I wasn't sending HTTP headers with the data just the raw data with a couple of Identifier bytes at the beginning to divert it into a logfile which I knew would come back to bite me. Bit of a black art that HTTP protocol.

Mike G

I'm not suggesting HTTP is the way to go - just an example. Consider sending the number of bytes in the message along with the Identifier bytes.

oodes

Cool , I can give that a go. Probably should use UDP like you suggested previously as the number of bytes would be in the UDP packet but barring this issue the system has been running fine for over a month( after changing to sending every command as it happens instead of trying to send logfiles of data as you suggested also) so I was reluctant to change too much.

oodes

Mike ,

I recently moved my set up (3 spinneret clients and a Spinneret server) to another location and another router and I have been experiencing some server crashes since I moved it. I done some logging and found some http GET's that seemed to have caused the problem.

Got  a connection
Bytes to Read is: 81
Buff Data is:
13/03/2014 17:40:11GET /phpTest/zologize/axa.php HTTP/1.1     // this here is what seems caused the problem
Host: 80.40.90.245
Connection: Close

I googled it and it appears that its malicious . I also had another strange one although it didnt crash the server

Got  a connection
Bytes to Read is: 75
Buff Data is:
13/03/2014 17:34:24GET /robots.txt HTTP/1.1
Host: 86.40.90.245
Accept-Encoding: identity

Any ideas on how to stop this? I had moved to port 80 to enable dyndns and had to switch the firewall security to low on the router, Can this be causing the problem with attempted hacks?

Mike G

Simple, return a "404 file not found" if the requested resource does not exist.

robots.txt is a configuration file for web crawlers like Google

oodes

Thanks Mike.

Thats interesting about the robots.txt.