Need help from GCC masters: Questions because of hanging program and memory related

trancefreak

Hi,

I ran into a problem with a hanging (not working program). It's quite difficult to describe the issue without creating a long story, so I'll try to keep short but informative.
I've attached the zipped program converted to a simpleIDE project because I think most of the propGCC users use simpleIDE or have it installed at least.

The project is set up to be compiled in XMMC memory model. To test, a quickstart board is enough because the EEPROM is big enough to execute the program in XMMC mode.
Make sure to set up the XMM cache with 2kb because that is the setting I'm using.

IMPORTANT:
I'm using the propgcc default branch, not the one included with the simpleIDE binaries. Reason is because the default branch supports multiple cogs executing code in XMM model while the release branch just supports one cog (you cannot use cogstart - or it doesn't work - in the release branch).

What does the program:
Basically, it just starts up one cog with cogstart which executes code in a loop (write out "Scheduler COG running...\n" every 1/4 second).
The main cog also executes some code in a loop.

If you compile and execute the program, you will notice that it doesn't print out "Scheduler COG running...\n" which means the started cog hangs.
If you change line 49 of main.cpp from:

const uint32_t testIndex = 5000;

to

const uint32_t testIndex = 1000;

the program prints out the logline above.

The issue is, there should be way enough HUB RAM available to hold the test array with size of 5000 bytes.

If I use propeller-elf-objdump -h main.elf I get the following (with the 5000 bytes array in the code):

main.elf:     file format elf32-propeller

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .xmmkernel    00000688  00000000  e0000000  00009350  2**2
                  CONTENTS, ALLOC, LOAD, CODE
  1 .init         0000013c  30000000  30000000  0000196c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  2 .text         000070d8  3000013c  3000013c  00001aa8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  3 IODriver.cog  00000794  00000000  30007214  00008b80  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  4 .fini         0000003c  300079a8  300079a8  00009314  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 .hub          00001558  00000000  00000000  00000154  2**2
                  CONTENTS, ALLOC, LOAD, CODE
  6 .math.kerext  000000dc  000006c0  00001558  000016ac  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  7 .start.kerext 000000cc  000006c0  00001634  00001788  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  8 .float.kerext 000000fc  000006c0  00001700  00001854  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  9 .ctors        0000000c  000017fc  000017fc  00001950  2**2
                  CONTENTS, ALLOC, LOAD, CODE
 10 .dtors        00000010  00001808  00001808  0000195c  2**2
                  CONTENTS, ALLOC, LOAD, CODE
 11 .bss          00001e80  00001818  00001818  0000196c  2**2
                  ALLOC
 12 .hub_heap     00000004  00003698  00003698  0000196c  2**0
                  ALLOC
 13 .debug_aranges 00000100  00000000  00000000  000099d8  2**0
                  CONTENTS, READONLY, DEBUGGING
 14 .debug_info   000023fc  00000000  00000000  00009ad8  2**0
                  CONTENTS, READONLY, DEBUGGING
 15 .debug_abbrev 0000084c  00000000  00000000  0000bed4  2**0
                  CONTENTS, READONLY, DEBUGGING
 16 .debug_line   000006e8  00000000  00000000  0000c720  2**0
                  CONTENTS, READONLY, DEBUGGING
 17 .debug_frame  00000164  00000000  00000000  0000ce08  2**2
                  CONTENTS, READONLY, DEBUGGING
 18 .debug_str    00000014  00000000  00000000  0000cf6c  2**0
                  CONTENTS, READONLY, DEBUGGING
 19 .debug_loc    00001bc4  00000000  00000000  0000cf80  2**0
                  CONTENTS, READONLY, DEBUGGING
 20 .debug_ranges 00000280  00000000  00000000  0000eb44  2**0
                  CONTENTS, READONLY, DEBUGGING

As far as I understand, the HUB RAM relevant parts are:
.hub (??) and .bss (stack size).

If I sum up these values I get:
33D8h => 13272d

So 13272 bytes used by the program itself. Additionally I have to add two XMM caches a 2048 bytes (2k cache size). So the overall usage is about 17368.
This size is not 100% correct, there could be additional RAM usage, but I'm sure not in the x kb area. So I think the program should run (with the 5kb testArr size), but it doesn't.
Only reducing the value to at least 4000 bytes solves the issue.

??? Why can't I use the complete available HUB RAM which should be free according to propeller-elf-objdump? There should be more than 10kb left.

Hope some of you C/C++ masters or propgcc developers could help me out with some thoughts or ideas.

Thanks,
Christian

trancefreak

I know now what causes the hanging program but I don't have a clue why. It's the COGC driver.

This is the main program:

#include <propeller.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include "Game\PinballHSM.h"
#include "Domain\SchedulerRegistry.h"
#include "Lamps\LampShow.h"
#include "Domain\PlayfieldSwitchEnum.h"
#include "CogC_Drivers\IODriver\IODriverMailbox.h"

#define MAX_LOOP_COUNTER 1

extern "C" void __cxa_pure_virtual() {
  while (1);
}

extern _Driver _SimpleSerialDriver;
//extern _Driver _FullDuplexSerialDriver;
extern _Driver _FileDriver;

/* Driver list */
_Driver *_driverlist[] = {
    &_SimpleSerialDriver,
    &_FileDriver,
    NULL
};

static struct {
    uint32_t ioDriverStack[IO_DRIVER_STACK_SIZE];
    volatile ioDriverMailbox_t ioDriverMailbox;
} ioDriverPar;

// Global variables / objects
uint32_t _loopArray[MAX_LOOP_COUNTER];
uint8_t _loopArrayCounter = 0;
uint8_t switchKey = 0;
const uint8_t activated = 128;
SchedulerRegistry schedulerRegistry;
PinballHSM game;

// THIS IS THE PROBLEMATIC PART:
const uint32_t testIndex = 8000;
volatile uint8_t testArr[testIndex];
uint32_t hub_buffer[496];

void handleSwitches(uint8_t);

// Scheduler cog stack declaration
static uint8_t schedulerCogStack[EXTRA_STACK_BYTES + 500];       // allocate space for the stack - you need to set up a
                                                                  // stack for each Cog you plan on starting

/*
 * Function to start up a new cog running the IO driver
 * code (which we have placed in the IODriver.cog section)
 */
uint8_t startIODriver(volatile void *parptr) {
  extern unsigned int _load_start_IODriver_cog[];
  #if defined(__PROPELLER_XMM__) || defined(__PROPELLER_XMMC__)
    extern unsigned int _load_stop_IODriver_cog[];
    memcpy(hub_buffer, _load_start_IODriver_cog, (_load_stop_IODriver_cog - _load_start_IODriver_cog) * sizeof(uint32_t));
    return cognew(hub_buffer, parptr);
  #else
    return cognew(_load_start_IODriver_cog, parptr);
  #endif
}

void schedulerCog(void* par) {
  // Schedule tasks.
  while(1) {
    schedulerRegistry.schedule();
  }
}

int main (void) {
  // Start the IO driver cog.
  cogstart(schedulerCog, NULL, schedulerCogStack, sizeof(schedulerCogStack));
  waitcnt((CLKFREQ * 2) + CNT);
  startIODriver(&ioDriverPar.ioDriverMailbox);

  for (uint32_t y = 0; y < testIndex; y++) {
    testArr[y] = 1;
  }

  LampShow lampShow((uint8_t*) ioDriverPar.ioDriverMailbox.lampState, (uint8_t*) ioDriverPar.ioDriverMailbox.outputPort);
  game.init();
  schedulerRegistry.addScheduler(lampShow, 31);

  uint8_t previousSwitchInpurtPort[64];
  for (uint8_t loopCounter = 0; loopCounter < 64; loopCounter++) {
    // Initialize with value 2 which means, not closed and not open.
    previousSwitchInpurtPort[loopCounter] = 2;
  }

  int startCnt = CNT;
  int endCnt = CNT;
  lampShow.playLampShow(LampShow::Sequence::BallPlunged);

  while (1) {
    printf("In main loop...\n");
    waitcnt((CLKFREQ / 4) + CNT);
    startCnt = CNT;
    // Handle only switch change events.
    for (uint8_t i = 0; i < 64; i++) {
      if (previousSwitchInpurtPort[i] == ioDriverPar.ioDriverMailbox.inputPort[i]) {
        // No change happened.
        continue;
      }

      // Switch state changed, notify game state machine.
      previousSwitchInpurtPort[i] = ioDriverPar.ioDriverMailbox.inputPort[i];
      switchKey = i;

      if (ioDriverPar.ioDriverMailbox.inputPort[i] > 0) {   // Switch is activated.
        switchKey = i | activated;
      }

      // Now call the appropriate switch function.
      handleSwitches(switchKey);
    }

    endCnt = CNT;
  }
}

void handleSwitches(uint8_t switchKey) {
  switch (switchKey) {
    case (int)PlayfieldSwitch::ShooterLanePlungerSwitch :
      game.onShooterLanePlungerSwitchClosed();
      break;

    case (int)PlayfieldSwitch::ShooterLanePlungerSwitch | activated :
      game.onShooterLanePlungerSwitchOpened();
      break;

    case (int)PlayfieldSwitch::ShooterLaneRampSwitch :
      game.onShooterLaneRampSwitchClosed();
      break;

    case (int)PlayfieldSwitch::ShooterLaneRampSwitch | activated :
      game.onShooterLaneRampSwitchOpened();
      break;

    case (int)PlayfieldSwitch::OuterLoopRightSwitch :
      game.onOuterLoopRightSwitchClosed();
      break;

    case (int)PlayfieldSwitch::OuterLoopRightSwitch | activated :
      game.onOuterLoopRightSwitchOpened();
      break;

    case (int)PlayfieldSwitch::OuterLoopLeftSwitch :
      game.onOuterLoopLeftSwitchClosed();
      break;

    case (int)PlayfieldSwitch::OuterLoopLeftSwitch | activated :
      game.onOuterLoopLeftSwitchOpened();
      break;

    case (int)PlayfieldSwitch::ThingRampSwitch :
      game.onThingRampSwitchClosed();
      break;

    case (int)PlayfieldSwitch::ThingRampSwitch | activated :
      game.onThingRampSwitchOpened();
      break;

    case (int)PlayfieldSwitch::ThingHoleSwitch :
      game.onThingHoleSwitchClosed();
      break;

    case (int)PlayfieldSwitch::ThingHoleSwitch | activated :
      game.onThingHoleSwitchOpened();
      break;

    case (int)PlayfieldSwitch::FarLeftInlaneSwitch :
      game.onFarLeftInlaneSwitchClosed();
      break;

    case (int)PlayfieldSwitch::FarLeftInlaneSwitch | activated :
      game.onFarLeftInlaneSwitchOpened();
      break;

    case (int)PlayfieldSwitch::LeftOutlaneSwitch :
      game.onLeftOutlaneSwitchClosed();
      break;

    case (int)PlayfieldSwitch::LeftOutlaneSwitch | activated :
      game.onLeftOutlaneSwitchOpened();
      break;

    case (int)PlayfieldSwitch::InnerLeftInlaneSwitch :
      game.onLeftInnerInlaneSwitchClosed();
      break;

    case (int)PlayfieldSwitch::InnerLeftInlaneSwitch | activated :
      game.onLeftInnerInlaneSwitchOpened();
      break;

    case (int)PlayfieldSwitch::RightInlaneSwitch :
      game.onRightInlaneSwitchClosed();
      break;

    case (int)PlayfieldSwitch::RightInlaneSwitch | activated :
      game.onRightInlaneSwitchOpened();
      break;

    case (int)PlayfieldSwitch::RightOutlaneSwitch :
      game.onRightOutlaneSwitchClosed();
      break;

    case (int)PlayfieldSwitch::RightOutlaneSwitch | activated :
      game.onRightOutlaneSwitchOpened();
      break;

    case (int)PlayfieldSwitch::Bumper1Switch :
      game.onBumper1SwitchOpened();
      break;

    case (int)PlayfieldSwitch::Bumper1Switch | activated :
      game.onBumper1SwitchOpened();
      break;

    case (int)PlayfieldSwitch::Bumper2Switch :
      game.onBumper2SwitchClosed();
      break;

    case (int)PlayfieldSwitch::Bumper2Switch | activated :
      game.onBumper2SwitchOpened();
      break;

    case (int)PlayfieldSwitch::Bumper3Switch :
      game.onBumper3SwitchClosed();
      break;

    case (int)PlayfieldSwitch::Bumper3Switch | activated :
      game.onBumper3SwitchOpened();
      break;

    case (int)PlayfieldSwitch::Bumper4Switch :
      game.onBumper4SwitchClosed();
      break;

    case (int)PlayfieldSwitch::Bumper4Switch | activated :
      game.onBumper4SwitchOpened();
      break;

    case (int)PlayfieldSwitch::Bumper5Switch :
      game.onBumper5SwitchClosed();
      break;

    case (int)PlayfieldSwitch::Bumper5Switch | activated :
      game.onBumper5SwitchOpened();
      break;

    case (int)PlayfieldSwitch::LeftFlipperButtonSwitch :
      game.onLeftFlipperButtonSwitchClosed();
      break;

    case (int)PlayfieldSwitch::LeftFlipperButtonSwitch | activated :
      game.onLeftFlipperButtonSwitchOpened();
      break;

    case (int)PlayfieldSwitch::RightFlipperButtonSwitch :
      game.onRightFlipperButtonSwitchClosed();
      break;

    case (int)PlayfieldSwitch::RightFlipperButtonSwitch | activated :
      game.onRightFlipperButtonSwitchOpened();
      break;
  }
}

If I comment out the starting of the driver everything works reliable and as expected.

// startIODriver(&ioDriverPar.ioDriverMailbox);

For testing, I made the stack size for the COGC driver 1200 bytes, but this didn't solve the issue and the driver never needs this big stack size.
Interestingly, the driver works and runs while the main program hangs.

Does anyone have an idea what I could check to be sure that there are no stack issues or what could cause the hanging when the COGC driver is copied to a COG and started there.
I'm helpless, completely.

Christian

ersmith

Have you trued changing the IO driver COG to do nothing (just sit in a loop blinking an LED, for example)? The two possibilities are that the COG itself is causing a conflict, or that it's the process of starting the COG that's causing the issue (the memcpy into hub_buffer). You could also try commenting out the actual cognew call inside startIODriver rather than the whole call to startIODriver.

trancefreak

Hi Eric,

That's what I wanted to try out today evening when I'm at home, currently I'm at the office (Making a simple COGC which just blinks an LED).
First I completele want to strip down the main program to be as slim as possible then I will add the simple COGC driver.

Additionally I found out that it has nothing to do with the start of a second XMM cog. If I comment out the line

cogstart(schedulerCog, NULL, schedulerCogStack, sizeof(schedulerCogStack));

the main loop still loops for 2 or 3 times then hangs or doesn't loop at all. So no difference of having that line in or out.

Only calling the startIODriver stuff influences the program. The main loop doesn't get hang up if I lower the index of the testArr[] to 4000 or lower.
So I assume there is something weird going on with regards to stack allocation or something else.

I'll try this out today evening and post an update.

Thanks for chiming in! :-)

Christian

ersmith

trancefreak wrote: »

Additionally I found out that it has nothing to do with the start of a second XMM cog. If I comment out the line

cogstart(schedulerCog, NULL, schedulerCogStack, sizeof(schedulerCogStack));

the main loop still loops for 2 or 3 times then hangs or doesn't loop at all. So no difference of having that line in or out.

Christian

Hmmm, odd. One thing to watch out for is that cogstart() itself needs quite a lot of stack (2KB) in the caller, because it makes a copy of the current COG's memory to use to start the new C cog. This isn't required for COGC code, just for running XMM/LMM in another COG.

Regards,
Eric

trancefreak

Hi Eric,

I played several hours now but I don't get what's going on. I've stripped the code quite down and have attached the project as simpleIDE project. Maybe you can do a quick test with a quickstart board.
If I comment out the line startIODriver(&ioDriverPar.ioDriverMailbox); the schedulerCog prints the message in a loop and everything works.

As soon as I uncomment the startIODriver line, the schedulerCog hangs. The COGC driver which is started with startIODriver(&ioDriverPar.ioDriverMailbox); is a completely stripped down COGC program which just blinks LED on pin 16 of the quickstart board.

If you could give the program a try maybe you are able to find out what's going wrong, I think I'm not able to find out why propgcc reacts so strange. There is enough HUB ram available so that cannot be the problem.
Looks like somehow there could be a stack conflict or something similar.

Regards,
Christian

ersmith

I haven't had a chance to test your program yet (I need to set up SimpleIDE on a new machine). But since it's quite stripped down now, could you try running it in LMM or CMM mode to see if it behaves any differently?

trancefreak

Hi Eric,

I think I found out what causes the problem. It looks like cognew overwrites memory if executed in xmmc mode.
In CMM mode everything is OK.

I added two functions which fill the stackarea of the cog driver with the pattern 0xAABB and check a handed over array for this pattern.
If I execute the program in CMM mode the stack area does not get changed and still holds the filled pattern. This is OK because the code just blinks an LED, so no stack is used.
If I execute the program in XMMC mode, the stack area gets overwritten with 0. I don't know how much is overwritten. I checked 6000 Bytes but didn't go further.

I tested the code with the release branch of propgcc and there the problem does not exist. Please could you check if there is an issue in the default branch when using xmmc mode and cognew.

I attached the code if you want to test.

Regards,
Christian

trancefreak

Update:

I tried the propgcc Windows build from SwimDude0614 (found in this thread: http://forums.parallax.com/showthread.php/160431-propgcc-now-in-the-Parallax-github) because this build is up to date.
But the problem with overwriting Hub RAM still exists. It does not have anything to do with using cogstart to start a new XMM Kernel in a new COG. The RAM get's corrupted when cognew is used.

Some variables have a new value after cognew because of the memory corruption.

@David Betz or Eric:
Do you have an idea why cognew damages the RAM? My test project is attached one message above.

Would be great if one of you could find the issue :-)


Regards,
Christian

ersmith

It looks like the C startup code in the COGC file is clearing the bss section (the section that holds uninitialized variables like your arrays). I don't quite understand why this should happen in XMMC mode but not CMM mode -- presumably there's some flag being set by the CMM kernel that the XMMC kernel is missing.

As a work around I suggest starting your COGC file as early as possible. and then telling it to wait for some flag to become non-zero before it continues (if it needs some initialization data from the host).

Thanks for the bug report!

ersmith

The flag that's not being set is a bit in the _C_LOCK variable, so an even simpler work-around is to add:

  extern int _C_LOCK;
  ...
  _C_LOCK |= 0x100;

somewhere in your startup code, before you try to launch a COGC file. This should make the CMM and XMMC versions work the same.

trancefreak

Hi Eric,

Thanks for taking the time to have a look! I will try to use the workaround. As long as no code section gets overwritten this should be fine for me.

Regards,
Christian

trancefreak

Hi Eric,

Where is the _C_LOCK variable defined? If I try to set the _C_LOCK |= 0x100 I get the following compiler error:

C:\Users\Christian\Documents\Pinball\Addams Family\Code\PinballAddamsFamily - Testwiese\Master\main.cpp|16|error: '_C_LOCK' does not name a type|

Regards,
Christian

ersmith

_C_LOCK is not declared anywhere -- you have to provide an "extern int _C_LOCK;" definition (the first line of the code snippet I posted) before you try to set it. (It is defined in the startup files, but that definition isn't made visible in the headers.)

trancefreak

I had the extern int _C_LOCK in place but it looks I have to move this code part into a function or within main.
I had it that way when the error message came up:

#include <propeller.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include "Domain\SchedulerRegistry.h"
#include "Lamps\LampShow.h"
#include "CogC_Drivers\IODriver\IODriverMailbox.h"

#define MAX_LOOP_COUNTER 1

extern "C" void __cxa_pure_virtual() {
  while (1);
}

extern int _C_LOCK;
_C_LOCK |= 0x100;

extern _Driver _SimpleSerialDriver;
//extern _Driver _FileDriver;

/* Driver list */
_Driver *_driverlist[] = {
    &_SimpleSerialDriver,
//    &_FileDriver,
    NULL
};

If I move the code snippet into the function which uses cognew, it works:

uint8_t startIODriver(volatile void *parptr) {
  extern int _C_LOCK;
  _C_LOCK |= 0x100;

  extern unsigned int _load_start_IODriver_cog[];
  #if defined(__PROPELLER_XMM__) || defined(__PROPELLER_XMMC__)
    char* heapBuffer = (char*) malloc(2000);
    if (heapBuffer == nullptr) {
      printf("Unable to allocate IODriver hub memory...\n");
      return 0;
    }
    extern unsigned int _load_stop_IODriver_cog[];
    memcpy(heapBuffer, _load_start_IODriver_cog, (_load_stop_IODriver_cog - _load_start_IODriver_cog) * sizeof(uint32_t));
    uint8_t cogId = cognew(heapBuffer, parptr);
    free(heapBuffer);
    return cogId;
  #else
    return cognew(_load_start_IODriver_cog, parptr);
  #endif
}

I'm going to play a bit tomorrow and test if it fixes the problem.
Thank you very much, Eric!

Regards,
Christian

trancefreak

Did the quick test just now and it works :-)
Stack no longer gets overwritten.