So, I have great news and bad news about the W6100.
First, the Bad. The UDP performance is terrible at 100Mbps. It is simply incapable of caching UDP data long enough, for the data to be pulled from the chip before the next packet arrives. My test was at 1.5Mbps of E1.31 data. (12 DMX universes (638 bytes + 6 header bytes) at 25Hz refresh) This is what it looks like from the perspective of the Interrupt line on the W6100 that goes low when data is available:
Yuck! You can see it is only receiving 2 of those universes and it appears that one of them has corrupt or missing data based on the fact they are not symmetrical. Indeed, I was only able to decode 2 of 12 universes tops at a 25Hz refresh rate, and the data was corrupt. This is even when configured with a 4K buffer.
This can be overcome by simply negotiating 10Mbps.
After testing at 10Mbps, I could see that the received data amounts were appropriate, however data was still often corrupt. The interrupts were not symmetrical. At this point I started to research using the W5300 on the Wiznet forums. In that research, I came across a user that had an issue exactly like mine. They were using the SPI interface, but were doing the same thing I was to determine if a packet is available: Checking the Sn_RX_RXR register. This register is 0 when no data is available and returns the size of the packet in bytes when data is in the buffer. The problem is, this register is populated as the data comes in, even before the end of packet. The Int line also goes low upon a non-0 number in Sn_RX_RXR. Wiznet instructed the user to check RXR twice, and if they were the same value, you have received a complete packet. That is what I did, and look at this glorious symmetry on the W6100 interrupt line:
We are cookin' with gas folks!! 12 DMX universes placed into hubRAM in 6.5ms. I am over the moon, because I now know it is possible to do 10's of thousands of pixels at a reasonable refresh rate, with this ethernet hardware.
I promise to release some code soon. It is coming, please be patient.
If anyone still cares, I laid out and had made a few boards for the WizNet 610. I don't know if they're still useful, given the work being done on the RMII interface, but hey. They exist. Ask nicely if you want one... S.
Quick update. I am now enumerating the E131 universe and only pulling the needed information from the packet. This has greatly reduced the amount of time it takes to read the needed data from the W6100. Now these were exceptionally short DMX universes with just 50 pixels, but it is 12 of them. The overall time has not changed, as the data is limited to 25 updates per second.
A low signal indicates there is data in the buffer available to be read.
My code is not ready for prime time. It is messy, horrible and I simply am "making it work" and moving on so as to keep some forward momentum with my light controller project. I am happy to say that I have a way to map multiple E1.31 universes into the pixel buffer. No fancy interface, but it works. So as not to be a code hoarder, I made my project public in Azure DevOps. At some point I will make a standalone web server. Please feel free to laugh at my jumbled mess of code I give you the work in progress of the GriswoldFX: Blitzen 24 light controller. https://griswoldfx.visualstudio.com/Blitzen%2024
@Scroungre I didn't have all the parts I thought I had to populate the connectors on the boards. Parts arrived late yesterday. I will try to work it this evening.
@Scroungre Bad news The data lines are in reverse order. D0 is connected to D7 on the Prop board, and they cross all the way to D7 on the Wiz610 connecting to D0 on the prop. There may be a software fix for that, but there are several places that needs to be updated, and we lose and instruction cycle. It is marked wrong on the schematic as well. D0 on the Wiznet is marked at D7 on the scmatic. JP4 Pin 10 is D0 and should be connected to
I assume the Wiz610 should be mounted from below, also JP1 and JP2? It's the only way the power lines aligned.
Yeah, something's definitely backwards there. I had in mind putting the WizNet in on top - but if the power's backwards for that, it's that which is wrong. I'll have a think about it and get back to you. Note that the KISS version will have exactly the same problems.
Grumble. But thanks much for the testing! I'll get back on this. S.
Pretty sure what happened was that I got the connectors on the WizNet module swapped. What they call JP1 I called JP2, and vice-versa. Okay, all those boards are trash - will re-do them and try again. sighS.
They're not exactly clearly marked in the spec sheet. Humph.
Update: I spent some time this week and have the primordial http server sending a page from the sdcard. I need to parse the http request to call this thing a real http server. That's up next on my list. From there I can configure it to send various MIME types, based on the requested filetype and have it serve up images and whatnot from the SD card.
I still need to build a locking mechanism so that everyone gets a chance at accessing the W6100. Since I want to prioritize access to it, I think locks would be better than a mailbox. Each process (cog) can determine how often it wants to access it, just by adding waits.
index.htm is a simple html file that prints some text and pulls a jpeg from the server. It and the jpeg file are located on an SD card plugged into the P2 Edge.
I even have a 404 page working, though is it very plain.
index.htm is a simple html file that prints some text and pulls a jpeg from the server. It and the jpeg file are located on an SD card plugged into the P2 Edge.
I even have a 404 page working, though is it very plain.
Hi, it worked fine, then I read "bounds checking" Sorry, I just coulnd't resist sending a very long url to your IP address to see if it was that simple.
You probably want to check for "invalid characters" as well as a "size" check. (for a quick and dirty fix, I'd guess the old spinneret code for the P1 would have to contain some simple filtering methods)
Look forward to the code
-H
@H said:
Hi, it worked fine, then I read "bounds checking" Sorry, I just coulnd't resist sending a very long url to your IP address to see if it was that simple.
You probably want to check for "invalid characters" as well as a "size" check. (for a quick and dirty fix, I'd guess the old spinneret code for the P1 would have to contain some simple filtering methods)
Look forward to the code
-H
H, I have not been able to reproduce the problem. Can you send me the request that brought it down?
Tried reproducing for a good 15 minutes, no luck, send it a lot of Smile lrepeating dots colons slashes and other special charcters that chrome doesnt escape to a hex sequence (ex. %7D) just to attempt breaking any tokenization or make a mess if something is passed unfiltered to a filesystem, can't seem to break it by simple means / browser. hammering with F5. seems solid for a casual attempt at vandalism, Done something to fix it?
@H Yes, I think I found the problem. The receive packet buffer in the P2 was actually smaller than the packet buffer in the chip. I watched your requests come in live
The long hang can happen if you hammer the system pretty hard. There is only one available socket and it can get into a state where it hangs where a FIN is not sent back from the client.
I have it configured so that if a connection comes in without an immediate request, such as a browser that wants a persistent connection, it will batt that connection down. That's the only way I have been able to get it to work with Chrome/Edge. There were lots of complaints about that on the Wiznet forums. The workaround was easy. I seem to recall having a similar problem with the Spinneret example code, when mine was still working.
OK, I found it. If the request was too long, it was truncated, as expected, however, it was still looking for a filename. I was using the location of "HTTP/1.1" to determine the end of the filename. It would return a negative number if it was not found. This would in turn cause the midstr function to write in parts of memory that I didn't want it to. Added a simple sanity check and it looks like I can't bring it down. I am going to leave it up for a while and see if it falls over again. Pretty sure I was getting randos on the net trying buffer overflows, so good chance if it stays up, it's OK.
I have implemented locks in the W6100.spin2 file and added the webserver method to test.spin2. It appears that the locking allows the two cogs to coexist, accessing that resource nicely. (Webserver and E1.31 receiver) In order for it to work properly I had to ensure that the pins for the W6100 were left in a floating state before releasing the lock.
Again, all of this code will need to be cleaned up. It's a freaking mess. I know I am creating technical debt by continuing forward without cleanup, but I want to keep the velocity moving forward. Christmas is coming and will be here sooner than I think
The locks work superbly! I found a bug that only shows itself at 10mbit. In order to do E1.31 the W6100 must be connected at 10Mbps. I put in a stop-gap patch and will update the W6100.spin2 in the repo. I will log the bug and prioritize it.
Here is a video of the webserver and E1.31 working at the same time. They are cooperatively accessing the W6100 chip from two different cogs. No need for mailboxes.
Work is under way for management via web browser. I now have dynamic form population via javascript includes working. I'm going old school. No frameworks, no middleware. 100% Javascript baby!
Here is the code that is called when net.js is requested from the server. I'm tickled pink it worked so well.
pub netjs()
bytefill(@webbuff, 0, 1000)
str.append_str(@webbuff,resp_ptr)
' set checkbox if DHCP is enabled
if DHCP == 1
str.append_str(@webbuff, dhcp_ptr)
' Set IP Address in form
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS0').value='"))
str.append_str(@webbuff,str.dec(IPADDRESS[0],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS1').value='"))
str.append_str(@webbuff,str.dec(IPADDRESS[1],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS2').value='"))
str.append_str(@webbuff,str.dec(IPADDRESS[2],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS3').value='"))
str.append_str(@webbuff,str.dec(IPADDRESS[3],-1))
str.append_str(@webbuff, string("';"))
' Set Subnet in form
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK0').value='"))
str.append_str(@webbuff,str.dec(SUBNETMASK[0],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK1').value='"))
str.append_str(@webbuff,str.dec(SUBNETMASK[1],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK2').value='"))
str.append_str(@webbuff,str.dec(SUBNETMASK[2],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK3').value='"))
str.append_str(@webbuff,str.dec(SUBNETMASK[3],-1))
str.append_str(@webbuff, string("';"))
' Set Gateway in form
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY0').value='"))
str.append_str(@webbuff,str.dec(GATEWAY[0],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY1').value='"))
str.append_str(@webbuff,str.dec(GATEWAY[1],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY2').value='"))
str.append_str(@webbuff,str.dec(GATEWAY[2],-1))
str.append_str(@webbuff, string("';"))
str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY3').value='"))
str.append_str(@webbuff,str.dec(GATEWAY[3],-1))
str.append_str(@webbuff, string("';"))
rdsize := strsize(@webbuff)
Comments
So, I have great news and bad news about the W6100.
First, the Bad. The UDP performance is terrible at 100Mbps. It is simply incapable of caching UDP data long enough, for the data to be pulled from the chip before the next packet arrives. My test was at 1.5Mbps of E1.31 data. (12 DMX universes (638 bytes + 6 header bytes) at 25Hz refresh) This is what it looks like from the perspective of the Interrupt line on the W6100 that goes low when data is available:
Yuck! You can see it is only receiving 2 of those universes and it appears that one of them has corrupt or missing data based on the fact they are not symmetrical. Indeed, I was only able to decode 2 of 12 universes tops at a 25Hz refresh rate, and the data was corrupt. This is even when configured with a 4K buffer.
This can be overcome by simply negotiating 10Mbps.
After testing at 10Mbps, I could see that the received data amounts were appropriate, however data was still often corrupt. The interrupts were not symmetrical. At this point I started to research using the W5300 on the Wiznet forums. In that research, I came across a user that had an issue exactly like mine. They were using the SPI interface, but were doing the same thing I was to determine if a packet is available: Checking the Sn_RX_RXR register. This register is 0 when no data is available and returns the size of the packet in bytes when data is in the buffer. The problem is, this register is populated as the data comes in, even before the end of packet. The Int line also goes low upon a non-0 number in Sn_RX_RXR. Wiznet instructed the user to check RXR twice, and if they were the same value, you have received a complete packet. That is what I did, and look at this glorious symmetry on the W6100 interrupt line:
We are cookin' with gas folks!! 12 DMX universes placed into hubRAM in 6.5ms. I am over the moon, because I now know it is possible to do 10's of thousands of pixels at a reasonable refresh rate, with this ethernet hardware.
I promise to release some code soon. It is coming, please be patient.
If anyone still cares, I laid out and had made a few boards for the WizNet 610. I don't know if they're still useful, given the work being done on the RMII interface, but hey. They exist. Ask nicely if you want one...
S.
Um, yes please!!!! I would like to take one of each! PM me how I can paypal you or otherwise send you money! These are GREAT!
Quick update. I am now enumerating the E131 universe and only pulling the needed information from the packet. This has greatly reduced the amount of time it takes to read the needed data from the W6100. Now these were exceptionally short DMX universes with just 50 pixels, but it is 12 of them. The overall time has not changed, as the data is limited to 25 updates per second.
A low signal indicates there is data in the buffer available to be read.
My code is not ready for prime time. It is messy, horrible and I simply am "making it work" and moving on so as to keep some forward momentum with my light controller project. I am happy to say that I have a way to map multiple E1.31 universes into the pixel buffer. No fancy interface, but it works. So as not to be a code hoarder, I made my project public in Azure DevOps. At some point I will make a standalone web server. Please feel free to laugh at my jumbled mess of code
I give you the work in progress of the GriswoldFX: Blitzen 24 light controller.
https://griswoldfx.visualstudio.com/Blitzen%2024
@Scroungre the boards arrived! Thank you for the kind letter. I will test this evening and post results here.
@Scroungre I didn't have all the parts I thought I had to populate the connectors on the boards. Parts arrived late yesterday. I will try to work it this evening.
Excellent. No big rush! If you think they're good, I'll post all the files. S.
@Scroungre Bad news
The data lines are in reverse order. D0 is connected to D7 on the Prop board, and they cross all the way to D7 on the Wiz610 connecting to D0 on the prop. There may be a software fix for that, but there are several places that needs to be updated, and we lose and instruction cycle. It is marked wrong on the schematic as well. D0 on the Wiznet is marked at D7 on the scmatic. JP4 Pin 10 is D0 and should be connected to
I assume the Wiz610 should be mounted from below, also JP1 and JP2? It's the only way the power lines aligned.
Yeah, something's definitely backwards there. I had in mind putting the WizNet in on top - but if the power's backwards for that, it's that which is wrong. I'll have a think about it and get back to you. Note that the KISS version will have exactly the same problems.
Grumble. But thanks much for the testing! I'll get back on this. S.
Pretty sure what happened was that I got the connectors on the WizNet module swapped. What they call JP1 I called JP2, and vice-versa. Okay, all those boards are trash - will re-do them and try again. sigh S.
They're not exactly clearly marked in the spec sheet. Humph.
Update: I spent some time this week and have the primordial http server sending a page from the sdcard. I need to parse the http request to call this thing a real http server. That's up next on my list. From there I can configure it to send various MIME types, based on the requested filetype and have it serve up images and whatnot from the SD card.
I still need to build a locking mechanism so that everyone gets a chance at accessing the W6100. Since I want to prioritize access to it, I think locks would be better than a mailbox. Each process (cog) can determine how often it wants to access it, just by adding waits.
Hey folks! I have a modest webserver up and running on the P2 with the W6100 for networking.
http://71.203.214.57/index.htm
index.htm is a simple html file that prints some text and pulls a jpeg from the server. It and the jpeg file are located on an SD card plugged into the P2 Edge.
I even have a 404 page working, though is it very plain.
http://71.203.214.57/nofile.bla
I will post code in a few days. It will be incorporated into my LED controller above.
Please hit the site and let me know if it works for you or if there is a problem.
Thanks!
--Terry
Worked for me using the latest version of Firefox running on my laptop running Windows 7. Pages loaded quickly, good job...
Hi
Yep worked for me too, win 10 home, edge browser.
came up incredibly quickly.
Dave
Hi Terry,
Came right up for me on my iPad.
Jim
Someone brought it down, and I think I know where the problem is. I need better bounds checking on requests.
I have restarted it.
Hi, it worked fine, then I read "bounds checking" Sorry, I just coulnd't resist sending a very long url to your IP address to see if it was that simple.
You probably want to check for "invalid characters" as well as a "size" check. (for a quick and dirty fix, I'd guess the old spinneret code for the P1 would have to contain some simple filtering methods)
Look forward to the code
-H
Worked wonderfully for me just now! Well done!!!!
Both URLs working. No hacking. Just clicking the links. Mint 20.3, latest Opera.
H, I have not been able to reproduce the problem. Can you send me the request that brought it down?
it was just a couple of attempts and then it timed out
Thought I crashed it with a very long url in my chrome browser, just repeating the same url with copy paste a few 100 times, hoping for a buffer overflowing.. Now I'm doubting it was my "fault"
example (http://71.203.214.57//index.htmlhttp://71.203.214.57//index.htmlhttp://71.203.214.57//index.htmlhttp://71.203.214.57//index.htmlhttp://71.203.214.57//index.htmlhttp://71.203.214.57//index.htmlhttp://71.203.214.57//index.htmlhttp://71.203.214.57//index.html)
Tried reproducing for a good 15 minutes, no luck, send it a lot of Smile lrepeating dots colons slashes and other special charcters that chrome doesnt escape to a hex sequence (ex. %7D) just to attempt breaking any tokenization or make a mess if something is passed unfiltered to a filesystem, can't seem to break it by simple means / browser. hammering with F5. seems solid for a casual attempt at vandalism, Done something to fix it?
addit: thought I got it as it responded with timeout when i tried http://71.203.214.57/index.htm/https://https://71.203.214.57/www.DOMAIN.com
but, refresh and i get the "404"
@H Yes, I think I found the problem. The receive packet buffer in the P2 was actually smaller than the packet buffer in the chip. I watched your requests come in live
The long hang can happen if you hammer the system pretty hard. There is only one available socket and it can get into a state where it hangs where a FIN is not sent back from the client.
I have it configured so that if a connection comes in without an immediate request, such as a browser that wants a persistent connection, it will batt that connection down. That's the only way I have been able to get it to work with Chrome/Edge. There were lots of complaints about that on the Wiznet forums. The workaround was easy. I seem to recall having a similar problem with the Spinneret example code, when mine was still working.
Nope, that wasn't it. It is something else.
OK, I found it. If the request was too long, it was truncated, as expected, however, it was still looking for a filename. I was using the location of "HTTP/1.1" to determine the end of the filename. It would return a negative number if it was not found. This would in turn cause the midstr function to write in parts of memory that I didn't want it to. Added a simple sanity check and it looks like I can't bring it down. I am going to leave it up for a while and see if it falls over again. Pretty sure I was getting randos on the net trying buffer overflows, so good chance if it stays up, it's OK.
http://71.203.214.57/index.htm
After more than 24 hours with no crashes, I am going to take the webserver off line this evening.
I have implemented locks in the W6100.spin2 file and added the webserver method to test.spin2. It appears that the locking allows the two cogs to coexist, accessing that resource nicely. (Webserver and E1.31 receiver) In order for it to work properly I had to ensure that the pins for the W6100 were left in a floating state before releasing the lock.
Again, all of this code will need to be cleaned up. It's a freaking mess. I know I am creating technical debt by continuing forward without cleanup, but I want to keep the velocity moving forward. Christmas is coming and will be here sooner than I think
technical debt,
never heard but describes it nicely
Mike
The locks work superbly! I found a bug that only shows itself at 10mbit. In order to do E1.31 the W6100 must be connected at 10Mbps. I put in a stop-gap patch and will update the W6100.spin2 in the repo. I will log the bug and prioritize it.
Here is a video of the webserver and E1.31 working at the same time. They are cooperatively accessing the W6100 chip from two different cogs. No need for mailboxes.
Work is under way for management via web browser. I now have dynamic form population via javascript includes working. I'm going old school. No frameworks, no middleware. 100% Javascript baby!
Here is the code that is called when net.js is requested from the server. I'm tickled pink it worked so well.
pub netjs() bytefill(@webbuff, 0, 1000) str.append_str(@webbuff,resp_ptr) ' set checkbox if DHCP is enabled if DHCP == 1 str.append_str(@webbuff, dhcp_ptr) ' Set IP Address in form str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS0').value='")) str.append_str(@webbuff,str.dec(IPADDRESS[0],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS1').value='")) str.append_str(@webbuff,str.dec(IPADDRESS[1],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS2').value='")) str.append_str(@webbuff,str.dec(IPADDRESS[2],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('IPADDRESS3').value='")) str.append_str(@webbuff,str.dec(IPADDRESS[3],-1)) str.append_str(@webbuff, string("';")) ' Set Subnet in form str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK0').value='")) str.append_str(@webbuff,str.dec(SUBNETMASK[0],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK1').value='")) str.append_str(@webbuff,str.dec(SUBNETMASK[1],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK2').value='")) str.append_str(@webbuff,str.dec(SUBNETMASK[2],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('SUBNETMASK3').value='")) str.append_str(@webbuff,str.dec(SUBNETMASK[3],-1)) str.append_str(@webbuff, string("';")) ' Set Gateway in form str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY0').value='")) str.append_str(@webbuff,str.dec(GATEWAY[0],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY1').value='")) str.append_str(@webbuff,str.dec(GATEWAY[1],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY2').value='")) str.append_str(@webbuff,str.dec(GATEWAY[2],-1)) str.append_str(@webbuff, string("';")) str.append_str(@webbuff, string($0D, $0A, "document.getElementById('GATEWAY3').value='")) str.append_str(@webbuff,str.dec(GATEWAY[3],-1)) str.append_str(@webbuff, string("';")) rdsize := strsize(@webbuff)