Reducing memory use SD card, sscanf, etc.

geo_leeman

Hi all - I'm doing one of those "worst case" projects for memory use on the prop. I'm getting a unit running that logs data from a serial instrument at ~10 Hz and writes it to an SD card. I'm checking that the received packet has the appropriate number of tabs (is at least roughly complete) and parsing it out in a packet struct that goes into a small ring buffer to prevent missed packets during SD block jumps (tested, and it does happen). I'm sitting at ~32180k sadly.

I think the main issues are in handle_state_write_data (line 108 TML.cpp), get_filename (line 240 TML.cpp), and read_new_data (line 52 ptm100.cpp). I've searched around on the forum, but didn't find a solution that appeared to help me much. Any ideas to help get this down a bit? I've got a bit of cleanup and minor implementation left to do, but it's pretty much all here now.

I switched to parsing the data out into the ptmpacket struct (ptm100.h) instead of just putting serial characters into the ring buffer to make it easier to only write full sentences, and maybe do some basic math on it in the future.

I've also attached a picture of the actual logging unit in case you're curious

Wuerfel_21

What a neat unit.
Are you already using CMM mode (as opposed to LMM)? (EDIT: It seems you do. The compiler command was inside the .side file)
If you have a 64k EEPROM, I think I recall there being an option to more-or-less automatically put cog images in the upper 32k and load those at runtime.

Cluso99

I don't really understand C so I've not looked at the code.
Is there anything that you could do in PASM to reduce the footprint, or use an existing (smaller) object(s)???

Dave Hein

I compiled your project, and the error message says that you overflowed by 32180 bytes. That means that the size is almost at 64K. This would take a lot of work to get it down below 32K. There are a few things you can do to reduce the size of your program, such as use CMM and the tiny lib. However, you are already doing that. Other things you can do are to eliminate formatted I/O by replacing them with you own smaller routines. It will be hard to shed 32K.

As Wuerfel_21 suggested, you could try breaking your program up into multiple programs, and store them in the upper portion of a 64K EEPROM.

Peter Jakacki

@geo_leeman - just so you know - I have done data logging to FAT32 cards for many years now and I can capture high speed serial streams continuously. Have I run out of space on the Prop? No, I even have Ethernet drivers and servers running and still have memory left over. So it is possible although C does not seem to be a good fit for a memory limited Prop.

geo_leeman

Ok - I thought overflowed by 32180 could not possibly be right... here's the strange part. If the function read_new_data in ptm100.cpp has its contents commented out, the compiled size is 28485 total with no errors... what on Earth is causing that single function to bloat the compiled size by > 2x?

DavidZemon

I'll dig into the code later - for now I'm just looking at your .side file. I see references to both fdserial and libtiny - that's not good if they're actually both getting used as they have a fair bit of overlap. You do have -enable_pruning, so if your code only references one of them then you should be fine.

All of the Simple objects are great, but some of them are more space efficient than others. Their primary objective is to be easy (think education) - not efficient.

Based on the filenames, I'm guessing you're using the SD driver shipped with PropGCC & Simple. You might try switching to libpropeller's SD class. No promises that it will be smaller, but it might be.
For once, I specifically do not recommend PropWare's SD/FAT library because it is actually a little bit larger than PropGCC + Simple's version (though many times faster).

DavidZemon

geo_leeman wrote: »

Ok - I thought overflowed by 32180 could not possibly be right... here's the strange part. If the function read_new_data in ptm100.cpp has its contents commented out, the compiled size is 28485 total with no errors... what on Earth is causing that single function to bloat the compiled size by > 2x?

It's not. It is an interesting coincidence though! But based on the fact that Dave Hein is recommending you switch to CMM, I'm guessing he compiled as LMM. I can see in your .side file that you're compiling as CMM already, and indeed, it does compile for me. I'm using a different compiler (built March of '17) and PropWare's build system, but without any code modifications I get the following:

Code size  = 28512
Total size = 31056

.

Pretty darned close to your numbers.

I see a lot of usage of functions with the name "print" and "scan" in them. Unfortunately, all of those are very heavy weight. For the print functions, there are usually printi alternatives. For instance, replacing dprint and sprintf with dprinti and sprinti should drop your code usage significantly. For the scan functions, try replacing them with your own scan function utilizing atoi. The reason all those print/scan functions are bloating your code size is because they include support for floating point types. PropGCC includes the special *i functions where the "i" stands for "integer" (presumably?), implying that these special functions exclude floating point support.

Post another version of your code and an update on your code size once you have that done and we can go from there if further changes are needed.

Dave Hein

The project is already using CMM, so that's not the issue. I also found if I put a "#if 0" around the contents of the read_new_data function the size is less than 32K. However, if I keep the line "static ptmpacket data_packet;" the program becomes very large again. It appears that the data_packet struct is very large. I looked at the contents of ptmpacket, and it doesn't seem that large, which implies that there is some C++ magic going on that is causing the size to blow up. Maybe a C++ guru could look at it and immediately see where the problem is.

You could try converting all the code to C, and maybe that would reduce the size.

DavidZemon

@"Dave Hein" , strange that you're getting such drastically different code size results from me. I know where not both using exactly the same setup... but they shouldn't be that different.

Also, forgive my suggestion of sscani... despite it being listed in the Simple header, there is no implementation for it (that PropWare's build system can find). If I try to build a program that uses it, I get a linker error. If I look through the Simple source code, no such file exists and I couldn't find the function in any similarly named files. I guess it's a bug in Simple? Should the implementation be added or the prototype removed from the header?

In any case, @geo_leeman, my recommendation stands: replace dprintf with dprinti and write your own scan function that utilizes atoi instead of sscanf. I've been working on your code to convert the sscanf function calls to use PropWare's Scanner class (mostly just for fun.. I don't expect you to use it...) and will let you know if that does anything for your code size soon. Once I'm done with that, I'll take a look at the rest of the code and see if there are any other low hanging fruit.

geo_leeman

Thanks for all of the recommendations - I've been working on this this morning and have some interesting results.

- Replace dprint with dprinti - done and saved about 4k
- Replace sprintf - actually make the memory situation worse
- Replace sscanf with atoi implementation. I commented out sscanf everywhere and saved some memory but that's not the biggest bloat.
- The struct is ~70 bytes. Changing the ring buffer size verifies this.

The biggest offender is.... static ptmpacket data_packet in the read_new_data function. If you take the static out the code drops to 21k total. If you move it out of the function to a global - same result. Any ideas there? Sure, moving it out solves the issue, but why is putting it in the method bloating things up so much?

DavidZemon

I wonder if you're seeing a compiler bug or something. When I move it out to a global, I get only an 8-byte difference is code size and 16-byte difference in total size.

DavidZemon

If I repeat Dave Hein's test of removing the entire contents of read_new_data, I get similarly unhelpful results, removing only about 500 bytes.

Dave Hein

I get the same results as geo_leeman. The program size is reduced if I move the declaration outside of the function. It seems like it could be a bug that may have been fixed by a later version of the tools.

geo_leeman

Is SimpleIDE the "recommended" dev environment? I'd personally prefer CLion or even Atom with tab completion, etc? Maybe that would let me use a more recent tool set? I'm on Windows if that makes any difference, but can of course get access to the Ubuntu subsystem as well.

Wuerfel_21

Any ol' editor that can edit C code should™ work. You just need to set it up to find the right compiler and libraries.

DavidZemon

If you prefer CLion and you're willing to use Linux, you should be using PropWare (ignore the Windows installation instructions... I don't think it's ever turned out well for anyone that's tried). Ubuntu is where I do all my development on PropWare so I can say with absolute certainty that it works very well there. And CLion is also where I do all of my development too . I even have instructions for creating CLion run configurations that build your program and then load it onto the Prop: https://david.zemon.name/PropWare/#/reference/using-an-ide
You'll need to download PropGCC separately. That's easy enough to get from here: https://david.zemon.name/PropWare/#/related-links

Keep in mind that using PropWare as your build system does not force you to use PropWare's objects. The installation packages include PropWare's classes as well as Simple, libpropeller, and libArduino. And to get you started, here's the CMake script I've been using to build your code:

cmake_minimum_required(VERSION 3.3)
find_package(PropWare 3.0 REQUIRED)

project(TML)

set(MODEL cmm)
create_simple_executable(TML
    TML.cpp

    DS3231.cpp
    cbuffer.cpp
    controlpanel.cpp
    datetime.cpp
    logging.cpp
    ptm100.cpp
    status.cpp)

I don't want to get this thread too far off topic though, so if you have any questions about how to install/use it, we can either take it over to the PropWare thread or privately via email or PM.

geo_leeman

Thanks @DavidZemon!

So yes, the end solution here is that it's a compiler bug and moving that struct from a static to a global solves the size issue.

I'm sitting at about 25-26k total now, which should be workable if I understand the 32k limitations. Not a ton of headroom, but some. Any guidance on the max advised?

DavidZemon

I don't remember if SimpleIDE reports a "total size" as well as the binary size or not. If it does, you can get more accurate numbers by moving as many variables as possible to the global space. For instance, take the following examples:

#include <PropWare/PropWare.h>
#include <PropWare/gpio/pin.h>

using PropWare::Pin;
using PropWare::Port;

int main () {
    const Pin led1(Port::P17, Pin::Dir::OUT);
    led1.start_hardware_pwm(4);

    const Pin led2(Port::P16, Pin::Dir::OUT);
    while (1) {
        led2.toggle();
        waitcnt(CLKFREQ / 4 + CNT);
    }
}

Results in:
Code size = 4368
Total size = 4540

But if I move those led variables to global:

#include <PropWare/PropWare.h>
#include <PropWare/gpio/pin.h>

using PropWare::Pin;
using PropWare::Port;

const Pin led1(Port::P17, Pin::Dir::OUT);
const Pin led2(Port::P16, Pin::Dir::OUT);

int main () {
    led1.start_hardware_pwm(4);

    while (1) {
        led2.toggle();
        waitcnt(CLKFREQ / 4 + CNT);
    }
}

Code size = 4392
Total size = 4588

I get a larger code size. Now, I haven't looked through the generated assembly, but I would bet this isn't any less efficient. I bet it's just letting the compiler do more static init and less at runtime.


Now, with that knowledge, I'll leave it to you to find an appropriate balance between knowing space requirements for your program and "globals are evil"