Strange behavior in hubexec mode

Dave Hein · 2015-10-21 02:03

I'm getting some inconsistent behavior with a program running in hubexec mode. When I add code to the program, or even just add a data buffer between two routines it starts to produce unexpected results. I checked the logic of my program, and it looks OK, but it's possible I have a bug in it. However, I found that if I add two NOPs after each CALL, JMP and RET the weirdness goes away. I wonder if there is something that gets out of sync for a few cycles when the streamer is restarted after a CALL/JMP/RET instruction. Has anybody else seen a problem like this?

I would post the code, but it's about 1000 lines of assembly. I haven't been able to reduce the program down to a few lines of code that still shows the problem. I'll keep playing with it to see if I can zero in on the specific instructions that are showing the problem.

Seairth · 2015-10-21 02:32

Are you switching between hub and cog exec mode at all, or is this entirely hub exec (all addresses >= $400)?

ozpropdev · 2015-10-21 03:53

I too have a very large program running hubexec and cogexec, with multiple cogs running.
I haven't seen behaviour like that.
I did however have a similar issue with P2-Hot that required inserting a NOP to fix a task glitch.

Dave Hein · 2015-10-21 13:19

My program does call multiply and divide routines that sit in cog memory, so it does switch back and forth between cog and hub exec modes. It uses only one cog. One thing I know for sure is that I can cause the problem to occur by putting a "long 0[100]" between my main routine and the rest of the routines. The other thing is that adding two NOPs after branches makes the problem go away. It may be that the problem is sensitive to the location of the code in memory, and it might have something to do with the alignment relative to the eggbeater. I'll run more tests today to see if I can narrow it down.

jmg · 2015-10-21 22:00

Dave Hein wrote: »

.... One thing I know for sure is that I can cause the problem to occur by putting a "long 0[100]" between my main routine and the rest of the routines. ..

It does sound possibly alignment related.
If you sweep that empty block in size, does the problem come/go matching the eggbeater modulus ?

Seairth · 2015-10-21 22:11

Are any of those instructions that follow the CALL an AUGS, AUGD, or ALTDS?

Dave Hein · 2015-10-22 02:57

I don't use AUGS, AUGD or ALTDS. However, I do use ## in some of my instructions, which would generate an AUGS. But that is never used after a CALL.

I think the affect of adding NOPs just moved code around in address space. I found that if I added lots of NOPs it could make the problem reappear, just like adding longs between routines.

I'm currently updating spinsim with the latest instruction changes, and I'm going to try running my program under it. If I don't see any strangeness under spinsim this would indicate there might be something wrong with the FPGA. If I do see the same thing under spinsim then the problem is in my code. If so, I will be able to debug it easier under spinsim than with the FPGA.

cgracey · 2015-10-22 05:03

There is probably some conceptual problem with my implementation of hub exec, from what is being described here. I suspect this is FIFO related.

If you could come up with some simplified example that can be made to work or fail with the addition or subtraction of NOPs, that would get me right to where I can diagnose it.

Dave Hein · 2015-10-29 02:09

I finally got spinsim updated so that it can execute code generated by the 10/10/15 version of PNut. I ran the code that fails on the FPGA and it works perfectly fine under spinsim. So it appears that there is some problem with hubexec in the FPGA.

As I was updating spinsim I ran into a problem with synchronization between the filling of the streaming FIFO and fetching code from it to be executed. It had something to do with both pieces of logic properly waiting for the correct eggbeater cycles. Maybe the FPGA has a similar problem.

I'll try running with the latest version of the FPGA to see if the problem persists. If the problem is still there I'll try to reduce my program to the minimum amount of code that shows the problem.

cgracey · 2015-10-29 03:12

Dave Hein wrote: »

I finally got spinsim updated so that it can execute code generated by the 10/10/15 version of PNut. I ran the code that fails on the FPGA and it works perfectly fine under spinsim. So it appears that there is some problem with hubexec in the FPGA.

As I was updating spinsim I ran into a problem with synchronization between the filling of the streaming FIFO and fetching code from it to be executed. It had something to do with both pieces of logic properly waiting for the correct eggbeater cycles. Maybe the FPGA has a similar problem.

I'll try running with the latest version of the FPGA to see if the problem persists. If the problem is still there I'll try to reduce my program to the minimum amount of code that shows the problem.

Dave, that would be great. I'm really anxious to fix any problem with hub exec.

78rpm · 2015-10-29 03:15

cgracey wrote: »

There is probably some conceptual problem with my implementation of hub exec, from what is being described here. I suspect this is FIFO related.

If you could come up with some simplified example that can be made to work or fail with the addition or subtraction of NOPs, that would get me right to where I can diagnose it.

Here is my modified test harness with very nearly a pattern of inserting longs and the error comes and goes. Eleven good ones and five with error messages. 11 + 5 = 16 Hmmmm. Good binary number, that

So you can reproduce this with the file attached. I use a DE0-Nano without add-on board, a Prop-Plug and Parallax Serial Terminal with all the preference tab special characters on, as I use a little cursor addressing

To find the code you need to go to xyzzy, it's a label and there are some comments. It's fine without any longs there, then the five-ish error ones, then eleven good.

I hope this helps. The routine where this is detected has been invoked via a calla / reta pair and it is always when the function returns the answer, so whether something gets mucked up on the call or on the ret or inbetween I'm not sure.

Of course, it could be my software...but even if it's the verilog, I'll still love you.

*** EDITED TO ADD ***

I forgot to say, it's 115,200 Baud and you need to hit Enter to kick it off - helps to have a wide screen, about 30-ish lines and clear the screen when you enable it. I'll add this last paragraph to my previous message
and I will also continue adding extra longs to see if this is a definite repeat. Oh, and monitor the forum.

78rpm · 2015-10-29 03:24

cgracey wrote: »

Dave Hein wrote: »

I finally got spinsim updated so that it can execute code generated by the 10/10/15 version of PNut. I ran the code that fails on the FPGA and it works perfectly fine under spinsim. So it appears that there is some problem with hubexec in the FPGA.

As I was updating spinsim I ran into a problem with synchronization between the filling of the streaming FIFO and fetching code from it to be executed. It had something to do with both pieces of logic properly waiting for the correct eggbeater cycles. Maybe the FPGA has a similar problem.

I'll try running with the latest version of the FPGA to see if the problem persists. If the problem is still there I'll try to reduce my program to the minimum amount of code that shows the problem.

Dave, that would be great. I'm really anxious to fix any problem with hub exec.

I forgot to say, it's 115,200 Baud and you need to hit Enter to kick it off - helps to have a wide screen, about 30-ish lines and clear the screen when you enable it. I'll add this last paragraph to my previous message
and I will also continue adding extra longs to see if this is a definite repeat. Oh, and monitor the forum.

78rpm · 2015-10-29 03:40

78rpm wrote: »

cgracey wrote: »

Dave Hein wrote: »

I finally got spinsim updated so that it can execute code generated by the 10/10/15 version of PNut. I ran the code that fails on the FPGA and it works perfectly fine under spinsim. So it appears that there is some problem with hubexec in the FPGA.

As I was updating spinsim I ran into a problem with synchronization between the filling of the streaming FIFO and fetching code from it to be executed. It had something to do with both pieces of logic properly waiting for the correct eggbeater cycles. Maybe the FPGA has a similar problem.

I'll try running with the latest version of the FPGA to see if the problem persists. If the problem is still there I'll try to reduce my program to the minimum amount of code that shows the problem.

Dave, that would be great. I'm really anxious to fix any problem with hub exec.

I forgot to say, it's 115,200 Baud and you need to hit Enter to kick it off - helps to have a wide screen, about 30-ish lines and clear the screen when you enable it. I'll add this last paragraph to my previous message
and I will also continue adding extra longs to see if this is a definite repeat. Oh, and monitor the forum.

I have done just over another two cycles of 16 longs, the pattern is 11 good ones, 1 with some errors, 3 with a lot of errors, 1 with some errors, and then the cycle repeats.

cgracey · 2015-10-29 03:57

78rpm wrote: »

78rpm wrote: »

cgracey wrote: »

Dave Hein wrote: »

I finally got spinsim updated so that it can execute code generated by the 10/10/15 version of PNut. I ran the code that fails on the FPGA and it works perfectly fine under spinsim. So it appears that there is some problem with hubexec in the FPGA.

As I was updating spinsim I ran into a problem with synchronization between the filling of the streaming FIFO and fetching code from it to be executed. It had something to do with both pieces of logic properly waiting for the correct eggbeater cycles. Maybe the FPGA has a similar problem.

I'll try running with the latest version of the FPGA to see if the problem persists. If the problem is still there I'll try to reduce my program to the minimum amount of code that shows the problem.

Dave, that would be great. I'm really anxious to fix any problem with hub exec.

I forgot to say, it's 115,200 Baud and you need to hit Enter to kick it off - helps to have a wide screen, about 30-ish lines and clear the screen when you enable it. I'll add this last paragraph to my previous message
and I will also continue adding extra longs to see if this is a definite repeat. Oh, and monitor the forum.

I have done just over another two cycles of 16 longs, the pattern is 11 good ones, 1 with some errors, 3 with a lot of errors, 1 with some errors, and then the cycle repeats.

You're using a DE0-Nano, right?

One thing I did that I never tested was to have the FIFO consider itself full 8 longs earlier on hub exec mode than during normal data mode, since hub exec eats data no faster than 1 long per two clocks. Perhaps, I need to up it by one.

If I made a DE0-Nano file without the early FIFO-full signal, could you see if it fixes the problem?

78rpm · 2015-10-29 03:59

cgracey wrote: »

78rpm wrote: »

78rpm wrote: »

cgracey wrote: »

Dave Hein wrote: »

I finally got spinsim updated so that it can execute code generated by the 10/10/15 version of PNut. I ran the code that fails on the FPGA and it works perfectly fine under spinsim. So it appears that there is some problem with hubexec in the FPGA.

As I was updating spinsim I ran into a problem with synchronization between the filling of the streaming FIFO and fetching code from it to be executed. It had something to do with both pieces of logic properly waiting for the correct eggbeater cycles. Maybe the FPGA has a similar problem.

I'll try running with the latest version of the FPGA to see if the problem persists. If the problem is still there I'll try to reduce my program to the minimum amount of code that shows the problem.

Dave, that would be great. I'm really anxious to fix any problem with hub exec.

I forgot to say, it's 115,200 Baud and you need to hit Enter to kick it off - helps to have a wide screen, about 30-ish lines and clear the screen when you enable it. I'll add this last paragraph to my previous message
and I will also continue adding extra longs to see if this is a definite repeat. Oh, and monitor the forum.

I have done just over another two cycles of 16 longs, the pattern is 11 good ones, 1 with some errors, 3 with a lot of errors, 1 with some errors, and then the cycle repeats.

You're using a DE0-Nano, right?

One thing I did that I never tested was to have the FIFO consider itself full 8 longs earlier on hub exec mode than during normal data mode, since hub exec eats data no faster than 1 long per two clocks. Perhaps, I need to up it by one.

If I made a DE0-Nano file without the early FIFO-full signal, could you see if it fixes the problem?

Yes DE0-Nano without the add-on board, I'll happily check for you, and PM if you want to do any of this by phone, not a problem.

Is Dave H using DE0-Nano?

cgracey · 2015-10-29 04:15

Ok, I'm going to make that special file now.

78rpm · 2015-10-29 04:20

OK, that's a quickish compile if you've done the modification, then time to put it somewhere for download. I'll be waiting ... patiently ... taps foot ...

cgracey · 2015-10-29 04:45

78rpm, here is that file:

DE0_Nano_Bare_Prop2_v2x1.zip

78rpm · 2015-10-29 04:48

cgracey wrote: »

78rpm, here is that file:

Loading now...

cgracey · 2015-10-29 04:58

If this doesn't fix the problem, I'll be wading in deep water soon.

Yanomani · 2015-10-29 05:01

Hi Chip

Will not Hubexec start loosing sync with next Fifo reload if we use RDLUT instructions?

cgracey · 2015-10-29 05:04

Yanomani wrote: »

Hi Chip

Will not Hubexec start loosing sync with next Fifo reload if we use RDLUT instructions?

No. That FIFO interface doesn't care what goes on.

I think it may be having a problem in hub exec mode, though, because I had it sense 'full' at 8 longs, instead of the normal 16, while the timing is a little advanced in hub exec because the first instruction is fetched right away. I changed the 'full' sensor to 9 longs now, so that 78rpm can see if it fixes the problem.

cgracey · 2015-10-29 05:07

78rpm, what's going on over there?

78rpm · 2015-10-29 05:07

There are still problems, but it is a lot better.

The group of 5 error longs has reduced by 1 and that good one is the first of the what were the 5 errors.

Previously for the 5 errors we had 1 okish, 3 bad, 1okish

Now we have 4, 1okish, 2 bad, 1 okish

So whatever you did is along the right tracks, but not quite there.

International Rescue thanks you for your custom and we now lay claim to Gracey Island.

cgracey · 2015-10-29 05:11

78rpm wrote: »

There are still problems, but it is a lot better.

The group of 5 error longs has reduced by 1 and that good one is the first of the what were the 5 errors.

Previously for the 5 errors we had 1 okish, 3 bad, 1okish

Now we have 4, 1okish, 2 bad, 1 okish

So whatever you did is along the right tracks, but not quite there.

International Rescue thanks you for your custom and we now lay claim to Gracey Island.

Ok, I'll woof up another file where there is no reduction in the 'full' sensor, so we can see if that completely fixes the problem. A little voice was telling me that this might not have been a good idea when I was implementing it.

Stand by...

cgracey · 2015-10-29 05:15

Ok. Try this...

DE0_Nano_Bare_Prop2_v2x2.zip

78rpm · 2015-10-29 05:18

cgracey wrote: »

78rpm wrote: »

There are still problems, but it is a lot better.

The group of 5 error longs has reduced by 1 and that good one is the first of the what were the 5 errors.

Previously for the 5 errors we had 1 okish, 3 bad, 1okish

Now we have 4, 1okish, 2 bad, 1 okish

So whatever you did is along the right tracks, but not quite there.

International Rescue thanks you for your custom and we now lay claim to Gracey Island.

Ok, I'll woof up another file where there is no reduction in the 'full' sensor, so we can see if that completely fixes the problem. A little voice was telling me that this might not have been a good idea when I was implementing it.

Stand by...

Post when ready, I'll put the kettle on... I think I've got RSI typying

    rep   @.until_mended, #patiently
    ",0"
    F11
    yes to save 
    click the PST
    enable
    clear
    try and position the cursor in the input line
    press enter
    inspect display 
.until_mended

    ' if you reach this point it means software errors are
    ' __YOUR__ responsibility. Yes, you, me, what, who?

    jmplnk #elsewhere   ' I've been paying attention and taking it in

And have a little stroll around the kitchen.

cgracey · 2015-10-29 05:22

I posted it above your last post, in case you missed it.

78rpm · 2015-10-29 05:25

cgracey wrote: »

I posted it above your last post, in case you missed it.

I have it loaded. Off we go, round 2...

78rpm · 2015-10-29 05:40

cgracey wrote: »

I posted it above your last post, in case you missed it.

I have had a fairly good test, it appears that not only have you solved the problem, buut you also get to keep Gracey Island. Well done, Chip!!!

Thank you, I enjoyed that interaction. It looks like I won't be needing that FINDBUG instruction now. Perhaps you could reallocate for Peter's jmplnk!

I have a large supply of commas and zeroes should you need some.

		alignl

' need to execute these with Parallax serial terminal, all special characters checked.
' make it's windo wide and about 30 lines high - easy to see when it goes wrong as 
' multiline space around message:
'  "Decimal error, digit not 0 - 9 ="
' and then the individual digits are printf out6 on the next line with some rather
' unusual ones. It always happens in this function, which uses a calla and reta.
' very nearly 11 good ones and 5 bad

xyzzy		long	0, 0, 0, 0, 0  ' perfect with just thes
     		long 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0		' error on decimal conversion
		long 	0, 0, 0, 0	' ditto on these
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0		' error again
		long    0, 0, 0 	' awfuul errors
		long	0		' small error
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0  ' these are good
		long	0		'	 error
'any longs after this line were added after I posted this file to the forum
		long       0, 0, 0, 0   ' errors
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long    0, 0, 0, 0, 0   ' errors
		long    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0, 0, 0, 0, 0	' errors but better
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0  ' ok
		long	0, 0, 0, 0
		long    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
' start back here v2
		long	0 'ok
		long	0 ' was error now ok
		long    0, 0 ' was big error
		long    0
		long    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0, 0, 0, 0, 0
		long	0, 0
' end of debugging the fpga image

cgracey · 2015-10-29 05:57

78rpm wrote: »

cgracey wrote: »

I posted it above your last post, in case you missed it.

I have had a fairly good test, it appears that not only have you solved the problem, buut you also get to keep Gracey Island. Well done, Chip!!!

Thank you, I enjoyed that interaction. It looks like I won't be needing that FINDBUG instruction now. Perhaps you could reallocate for Peter's jmplnk!

I have a large supply of commas and zeroes should you need some.

		alignl

' need to execute these with Parallax serial terminal, all special characters checked.
' make it's windo wide and about 30 lines high - easy to see when it goes wrong as 
' multiline space around message:
'  "Decimal error, digit not 0 - 9 ="
' and then the individual digits are printf out6 on the next line with some rather
' unusual ones. It always happens in this function, which uses a calla and reta.
' very nearly 11 good ones and 5 bad

xyzzy		long	0, 0, 0, 0, 0  ' perfect with just thes
     		long 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0		' error on decimal conversion
		long 	0, 0, 0, 0	' ditto on these
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0		' error again
		long    0, 0, 0 	' awfuul errors
		long	0		' small error
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0  ' these are good
		long	0		'	 error
'any longs after this line were added after I posted this file to the forum
		long       0, 0, 0, 0   ' errors
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long    0, 0, 0, 0, 0   ' errors
		long    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0, 0, 0, 0, 0	' errors but better
		long	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0  ' ok
		long	0, 0, 0, 0
		long    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
' start back here v2
		long	0 'ok
		long	0 ' was error now ok
		long    0, 0 ' was big error
		long    0
		long    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
		long	0, 0, 0, 0, 0
		long	0, 0
' end of debugging the fpga image

Whew! What a relief. Thank you for testing that. I had to go play a game of solitaire to get past the waiting.

I will make this change permanent now and may make an interim new set of files for the four target boards.

Strange behavior in hubexec mode

Comments