Reducing memory use SD card, sscanf, etc.

Hi all - I'm doing one of those "worst case" projects for memory use on the prop. I'm getting a unit running that logs data from a serial instrument at ~10 Hz and writes it to an SD card. I'm checking that the received packet has the appropriate number of tabs (is at least roughly complete) and parsing it out in a packet struct that goes into a small ring buffer to prevent missed packets during SD block jumps (tested, and it does happen). I'm sitting at ~32180k sadly.

I think the main issues are in handle_state_write_data (line 108 TML.cpp), get_filename (line 240 TML.cpp), and read_new_data (line 52 ptm100.cpp). I've searched around on the forum, but didn't find a solution that appeared to help me much. Any ideas to help get this down a bit? I've got a bit of cleanup and minor implementation left to do, but it's pretty much all here now.

I switched to parsing the data out into the ptmpacket struct (ptm100.h) instead of just putting serial characters into the ring buffer to make it easier to only write full sentences, and maybe do some basic math on it in the future.

I've also attached a picture of the actual logging unit in case you're curious :)
1261 x 1585 - 228K

Comments

  • 18 Comments sorted by Date Added Votes
  • Wuerfel_21Wuerfel_21 Posts: 173
    edited October 19 Vote Up0Vote Down
    What a neat unit.
    Are you already using CMM mode (as opposed to LMM)? (EDIT: It seems you do. The compiler command was inside the .side file)
    If you have a 64k EEPROM, I think I recall there being an option to more-or-less automatically put cog images in the upper 32k and load those at runtime.
  • Cluso99Cluso99 Posts: 14,165
    edited October 19 Vote Up0Vote Down
    I don't really understand C so I've not looked at the code.
    Is there anything that you could do in PASM to reduce the footprint, or use an existing (smaller) object(s)???
    My Prop boards: P8XBlade2, RamBlade, CpuBlade, TriBlade
    Prop OS (also see Sphinx, PropDos, PropCmd, Spinix)
    Website: www.clusos.com
    Prop Tools (Index) , Emulators (Index) , ZiCog (Z80)
  • I compiled your project, and the error message says that you overflowed by 32180 bytes. That means that the size is almost at 64K. This would take a lot of work to get it down below 32K. There are a few things you can do to reduce the size of your program, such as use CMM and the tiny lib. However, you are already doing that. Other things you can do are to eliminate formatted I/O by replacing them with you own smaller routines. It will be hard to shed 32K.

    As Wuerfel_21 suggested, you could try breaking your program up into multiple programs, and store them in the upper portion of a 64K EEPROM.
  • @geo_leeman - just so you know - I have done data logging to FAT32 cards for many years now and I can capture high speed serial streams continuously. Have I run out of space on the Prop? No, I even have Ethernet drivers and servers running and still have memory left over. So it is possible although C does not seem to be a good fit for a memory limited Prop.

    Tachyon Forth - compact, fast, forthwright and interactive
    useforthlogo-s.png
    --->CLICK THE LOGO for more links<---
    Latest binary V5.4 includes EASYFILE +++++ Tachyon Forth News Blog
    P2 SHORTFORM DATASHEET +++++ TAQOZ documentation
    Brisbane, Australia
  • Ok - I thought overflowed by 32180 could not possibly be right... here's the strange part. If the function read_new_data in ptm100.cpp has its contents commented out, the compiled size is 28485 total with no errors... what on Earth is causing that single function to bloat the compiled size by > 2x?
  • DavidZemonDavidZemon Posts: 2,514
    edited October 19 Vote Up0Vote Down
    I'll dig into the code later - for now I'm just looking at your .side file. I see references to both fdserial and libtiny - that's not good if they're actually both getting used as they have a fair bit of overlap. You do have -enable_pruning, so if your code only references one of them then you should be fine.

    All of the Simple objects are great, but some of them are more space efficient than others. Their primary objective is to be easy (think education) - not efficient.

    Based on the filenames, I'm guessing you're using the SD driver shipped with PropGCC & Simple. You might try switching to libpropeller's SD class. No promises that it will be smaller, but it might be.
    For once, I specifically do not recommend PropWare's SD/FAT library because it is actually a little bit larger than PropGCC + Simple's version (though many times faster).
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • geo_leeman wrote: »
    Ok - I thought overflowed by 32180 could not possibly be right... here's the strange part. If the function read_new_data in ptm100.cpp has its contents commented out, the compiled size is 28485 total with no errors... what on Earth is causing that single function to bloat the compiled size by > 2x?

    It's not. It is an interesting coincidence though! But based on the fact that Dave Hein is recommending you switch to CMM, I'm guessing he compiled as LMM. I can see in your .side file that you're compiling as CMM already, and indeed, it does compile for me. I'm using a different compiler (built March of '17) and PropWare's build system, but without any code modifications I get the following:
    Code size  = 28512
    Total size = 31056
    
    .

    Pretty darned close to your numbers.

    I see a lot of usage of functions with the name "print" and "scan" in them. Unfortunately, all of those are very heavy weight. For the print functions, there are usually printi alternatives. For instance, replacing dprint and sprintf with dprinti and sprinti should drop your code usage significantly. For the scan functions, try replacing them with your own scan function utilizing atoi. The reason all those print/scan functions are bloating your code size is because they include support for floating point types. PropGCC includes the special *i functions where the "i" stands for "integer" (presumably?), implying that these special functions exclude floating point support.

    Post another version of your code and an update on your code size once you have that done and we can go from there if further changes are needed.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • The project is already using CMM, so that's not the issue. I also found if I put a "#if 0" around the contents of the read_new_data function the size is less than 32K. However, if I keep the line "static ptmpacket data_packet;" the program becomes very large again. It appears that the data_packet struct is very large. I looked at the contents of ptmpacket, and it doesn't seem that large, which implies that there is some C++ magic going on that is causing the size to blow up. Maybe a C++ guru could look at it and immediately see where the problem is.

    You could try converting all the code to C, and maybe that would reduce the size.
  • @Dave Hein , strange that you're getting such drastically different code size results from me. I know where not both using exactly the same setup... but they shouldn't be that different.

    Also, forgive my suggestion of sscani... despite it being listed in the Simple header, there is no implementation for it (that PropWare's build system can find). If I try to build a program that uses it, I get a linker error. If I look through the Simple source code, no such file exists and I couldn't find the function in any similarly named files. I guess it's a bug in Simple? Should the implementation be added or the prototype removed from the header?

    In any case, @geo_leeman, my recommendation stands: replace dprintf with dprinti and write your own scan function that utilizes atoi instead of sscanf. I've been working on your code to convert the sscanf function calls to use PropWare's Scanner class (mostly just for fun.. I don't expect you to use it...) and will let you know if that does anything for your code size soon. Once I'm done with that, I'll take a look at the rest of the code and see if there are any other low hanging fruit.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • Thanks for all of the recommendations - I've been working on this this morning and have some interesting results.

    - Replace dprint with dprinti - done and saved about 4k
    - Replace sprintf - actually make the memory situation worse
    - Replace sscanf with atoi implementation. I commented out sscanf everywhere and saved some memory but that's not the biggest bloat.
    - The struct is ~70 bytes. Changing the ring buffer size verifies this.

    The biggest offender is.... static ptmpacket data_packet in the read_new_data function. If you take the static out the code drops to 21k total. If you move it out of the function to a global - same result. Any ideas there? Sure, moving it out solves the issue, but why is putting it in the method bloating things up so much?
  • I wonder if you're seeing a compiler bug or something. When I move it out to a global, I get only an 8-byte difference is code size and 16-byte difference in total size.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • If I repeat Dave Hein's test of removing the entire contents of read_new_data, I get similarly unhelpful results, removing only about 500 bytes.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • I get the same results as geo_leeman. The program size is reduced if I move the declaration outside of the function. It seems like it could be a bug that may have been fixed by a later version of the tools.
  • Is SimpleIDE the "recommended" dev environment? I'd personally prefer CLion or even Atom with tab completion, etc? Maybe that would let me use a more recent tool set? I'm on Windows if that makes any difference, but can of course get access to the Ubuntu subsystem as well.
  • Any ol' editor that can edit C code should™ work. You just need to set it up to find the right compiler and libraries.
  • If you prefer CLion and you're willing to use Linux, you should be using PropWare (ignore the Windows installation instructions... I don't think it's ever turned out well for anyone that's tried). Ubuntu is where I do all my development on PropWare so I can say with absolute certainty that it works very well there. And CLion is also where I do all of my development too :smile:. I even have instructions for creating CLion run configurations that build your program and then load it onto the Prop: https://david.zemon.name/PropWare/#/reference/using-an-ide
    You'll need to download PropGCC separately. That's easy enough to get from here: https://david.zemon.name/PropWare/#/related-links

    Keep in mind that using PropWare as your build system does not force you to use PropWare's objects. The installation packages include PropWare's classes as well as Simple, libpropeller, and libArduino. And to get you started, here's the CMake script I've been using to build your code:
    cmake_minimum_required(VERSION 3.3)
    find_package(PropWare 3.0 REQUIRED)
    
    project(TML)
    
    set(MODEL cmm)
    create_simple_executable(TML
        TML.cpp
    
        DS3231.cpp
        cbuffer.cpp
        controlpanel.cpp
        datetime.cpp
        logging.cpp
        ptm100.cpp
        status.cpp)
    

    I don't want to get this thread too far off topic though, so if you have any questions about how to install/use it, we can either take it over to the PropWare thread or privately via email or PM.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • Thanks @DavidZemon!

    So yes, the end solution here is that it's a compiler bug and moving that struct from a static to a global solves the size issue.

    I'm sitting at about 25-26k total now, which should be workable if I understand the 32k limitations. Not a ton of headroom, but some. Any guidance on the max advised?
  • I don't remember if SimpleIDE reports a "total size" as well as the binary size or not. If it does, you can get more accurate numbers by moving as many variables as possible to the global space. For instance, take the following examples:
    #include <PropWare/PropWare.h>
    #include <PropWare/gpio/pin.h>
    
    using PropWare::Pin;
    using PropWare::Port;
    
    int main () {
        const Pin led1(Port::P17, Pin::Dir::OUT);
        led1.start_hardware_pwm(4);
    
        const Pin led2(Port::P16, Pin::Dir::OUT);
        while (1) {
            led2.toggle();
            waitcnt(CLKFREQ / 4 + CNT);
        }
    }
    

    Results in:
    Code size = 4368
    Total size = 4540

    But if I move those led variables to global:
    #include <PropWare/PropWare.h>
    #include <PropWare/gpio/pin.h>
    
    using PropWare::Pin;
    using PropWare::Port;
    
    const Pin led1(Port::P17, Pin::Dir::OUT);
    const Pin led2(Port::P16, Pin::Dir::OUT);
    
    int main () {
        led1.start_hardware_pwm(4);
    
        while (1) {
            led2.toggle();
            waitcnt(CLKFREQ / 4 + CNT);
        }
    }
    

    Code size = 4392
    Total size = 4588

    I get a larger code size. Now, I haven't looked through the generated assembly, but I would bet this isn't any less efficient. I bet it's just letting the compiler do more static init and less at runtime.


    Now, with that knowledge, I'll leave it to you to find an appropriate balance between knowing space requirements for your program and "globals are evil" :D
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
Sign In or Register to comment.