Reducing memory use SD card, sscanf, etc.

Comments

• Wuerfel_21 Posts: 4,448

  2018-10-19 11:41 edited 2018-10-19 11:44

  What a neat unit.
  Are you already using CMM mode (as opposed to LMM)? (EDIT: It seems you do. The compiler command was inside the .side file)
  If you have a 64k EEPROM, I think I recall there being an option to more-or-less automatically put cog images in the upper 32k and load those at runtime.
• Cluso99 Posts: 18,069

  2018-10-19 11:52 edited 2018-10-19 11:53

  I don't really understand C so I've not looked at the code.
  Is there anything that you could do in PASM to reduce the footprint, or use an existing (smaller) object(s)???
• Dave Hein Posts: 6,347

  2018-10-19 12:08

  I compiled your project, and the error message says that you overflowed by 32180 bytes. That means that the size is almost at 64K. This would take a lot of work to get it down below 32K. There are a few things you can do to reduce the size of your program, such as use CMM and the tiny lib. However, you are already doing that. Other things you can do are to eliminate formatted I/O by replacing them with you own smaller routines. It will be hard to shed 32K.
  
  As Wuerfel_21 suggested, you could try breaking your program up into multiple programs, and store them in the upper portion of a 64K EEPROM.
• Peter Jakacki Posts: 10,193

  2018-10-19 12:33

  @geo_leeman - just so you know - I have done data logging to FAT32 cards for many years now and I can capture high speed serial streams continuously. Have I run out of space on the Prop? No, I even have Ethernet drivers and servers running and still have memory left over. So it is possible although C does not seem to be a good fit for a memory limited Prop.
• geo_leeman Posts: 190

  2018-10-19 13:41

  Ok - I thought overflowed by 32180 could not possibly be right... here's the strange part. If the function read_new_data in ptm100.cpp has its contents commented out, the compiled size is 28485 total with no errors... what on Earth is causing that single function to bloat the compiled size by > 2x?
• DavidZemon Posts: 2,973

  2018-10-19 15:05 edited 2018-10-19 15:13

  I'll dig into the code later - for now I'm just looking at your .side file. I see references to both fdserial and libtiny - that's not good if they're actually both getting used as they have a fair bit of overlap. You do have -enable_pruning, so if your code only references one of them then you should be fine.
  
  All of the Simple objects are great, but some of them are more space efficient than others. Their primary objective is to be easy (think education) - not efficient.
  
  Based on the filenames, I'm guessing you're using the SD driver shipped with PropGCC & Simple. You might try switching to libpropeller's SD class. No promises that it will be smaller, but it might be.
  For once, I specifically do not recommend PropWare's SD/FAT library because it is actually a little bit larger than PropGCC + Simple's version (though many times faster).
• DavidZemon Posts: 2,973

  2018-10-20 15:07

  geo_leeman wrote: »

  Ok - I thought overflowed by 32180 could not possibly be right... here's the strange part. If the function read_new_data in ptm100.cpp has its contents commented out, the compiled size is 28485 total with no errors... what on Earth is causing that single function to bloat the compiled size by > 2x?

  
  It's not. It is an interesting coincidence though! But based on the fact that Dave Hein is recommending you switch to CMM, I'm guessing he compiled as LMM. I can see in your .side file that you're compiling as CMM already, and indeed, it does compile for me. I'm using a different compiler (built March of '17) and PropWare's build system, but without any code modifications I get the following:
  

  Code size  = 28512
  Total size = 31056
  

  .
  
  Pretty darned close to your numbers.
  
  I see a lot of usage of functions with the name "print" and "scan" in them. Unfortunately, all of those are very heavy weight. For the print functions, there are usually printi alternatives. For instance, replacing dprint and sprintf with dprinti and sprinti should drop your code usage significantly. For the scan functions, try replacing them with your own scan function utilizing atoi. The reason all those print/scan functions are bloating your code size is because they include support for floating point types. PropGCC includes the special *i functions where the "i" stands for "integer" (presumably?), implying that these special functions exclude floating point support.
  
  Post another version of your code and an update on your code size once you have that done and we can go from there if further changes are needed.
• Dave Hein Posts: 6,347

  2018-10-21 00:53

  The project is already using CMM, so that's not the issue. I also found if I put a "#if 0" around the contents of the read_new_data function the size is less than 32K. However, if I keep the line "static ptmpacket data_packet;" the program becomes very large again. It appears that the data_packet struct is very large. I looked at the contents of ptmpacket, and it doesn't seem that large, which implies that there is some C++ magic going on that is causing the size to blow up. Maybe a C++ guru could look at it and immediately see where the problem is.
  
  You could try converting all the code to C, and maybe that would reduce the size.
• DavidZemon Posts: 2,973

  2018-10-21 05:22

  @"Dave Hein" , strange that you're getting such drastically different code size results from me. I know where not both using exactly the same setup... but they shouldn't be that different.
  
  Also, forgive my suggestion of sscani... despite it being listed in the Simple header, there is no implementation for it (that PropWare's build system can find). If I try to build a program that uses it, I get a linker error. If I look through the Simple source code, no such file exists and I couldn't find the function in any similarly named files. I guess it's a bug in Simple? Should the implementation be added or the prototype removed from the header?
  
  In any case, @geo_leeman, my recommendation stands: replace dprintf with dprinti and write your own scan function that utilizes atoi instead of sscanf. I've been working on your code to convert the sscanf function calls to use PropWare's Scanner class (mostly just for fun.. I don't expect you to use it...) and will let you know if that does anything for your code size soon. Once I'm done with that, I'll take a look at the rest of the code and see if there are any other low hanging fruit.
• geo_leeman Posts: 190

  2018-10-21 16:11

  Thanks for all of the recommendations - I've been working on this this morning and have some interesting results.
  
  - Replace dprint with dprinti - done and saved about 4k
  - Replace sprintf - actually make the memory situation worse
  - Replace sscanf with atoi implementation. I commented out sscanf everywhere and saved some memory but that's not the biggest bloat.
  - The struct is ~70 bytes. Changing the ring buffer size verifies this.
  
  The biggest offender is.... static ptmpacket data_packet in the read_new_data function. If you take the static out the code drops to 21k total. If you move it out of the function to a global - same result. Any ideas there? Sure, moving it out solves the issue, but why is putting it in the method bloating things up so much?
• DavidZemon Posts: 2,973

  2018-10-21 18:10

  I wonder if you're seeing a compiler bug or something. When I move it out to a global, I get only an 8-byte difference is code size and 16-byte difference in total size.
• DavidZemon Posts: 2,973

  2018-10-21 18:11

  If I repeat Dave Hein's test of removing the entire contents of read_new_data, I get similarly unhelpful results, removing only about 500 bytes.
• Dave Hein Posts: 6,347

  2018-10-21 20:35

  I get the same results as geo_leeman. The program size is reduced if I move the declaration outside of the function. It seems like it could be a bug that may have been fixed by a later version of the tools.
• geo_leeman Posts: 190

  2018-10-21 21:20

  Is SimpleIDE the "recommended" dev environment? I'd personally prefer CLion or even Atom with tab completion, etc? Maybe that would let me use a more recent tool set? I'm on Windows if that makes any difference, but can of course get access to the Ubuntu subsystem as well.
• Wuerfel_21 Posts: 4,448

  2018-10-21 21:24

  Any ol' editor that can edit C code should™ work. You just need to set it up to find the right compiler and libraries.
• DavidZemon Posts: 2,973

  2018-10-21 23:48

  If you prefer CLion and you're willing to use Linux, you should be using PropWare (ignore the Windows installation instructions... I don't think it's ever turned out well for anyone that's tried). Ubuntu is where I do all my development on PropWare so I can say with absolute certainty that it works very well there. And CLion is also where I do all of my development too . I even have instructions for creating CLion run configurations that build your program and then load it onto the Prop: https://david.zemon.name/PropWare/#/reference/using-an-ide
  You'll need to download PropGCC separately. That's easy enough to get from here: https://david.zemon.name/PropWare/#/related-links
  
  Keep in mind that using PropWare as your build system does not force you to use PropWare's objects. The installation packages include PropWare's classes as well as Simple, libpropeller, and libArduino. And to get you started, here's the CMake script I've been using to build your code:
  

  cmake_minimum_required(VERSION 3.3)
  find_package(PropWare 3.0 REQUIRED)
  
  project(TML)
  
  set(MODEL cmm)
  create_simple_executable(TML
      TML.cpp
  
      DS3231.cpp
      cbuffer.cpp
      controlpanel.cpp
      datetime.cpp
      logging.cpp
      ptm100.cpp
      status.cpp)
  

  
  I don't want to get this thread too far off topic though, so if you have any questions about how to install/use it, we can either take it over to the PropWare thread or privately via email or PM.
• geo_leeman Posts: 190

  2018-10-22 16:49

  Thanks @DavidZemon!
  
  So yes, the end solution here is that it's a compiler bug and moving that struct from a static to a global solves the size issue.
  
  I'm sitting at about 25-26k total now, which should be workable if I understand the 32k limitations. Not a ton of headroom, but some. Any guidance on the max advised?
• DavidZemon Posts: 2,973

  2018-10-22 17:26

  I don't remember if SimpleIDE reports a "total size" as well as the binary size or not. If it does, you can get more accurate numbers by moving as many variables as possible to the global space. For instance, take the following examples:
  

  #include <PropWare/PropWare.h>
  #include <PropWare/gpio/pin.h>
  
  using PropWare::Pin;
  using PropWare::Port;
  
  int main () {
      const Pin led1(Port::P17, Pin::Dir::OUT);
      led1.start_hardware_pwm(4);
  
      const Pin led2(Port::P16, Pin::Dir::OUT);
      while (1) {
          led2.toggle();
          waitcnt(CLKFREQ / 4 + CNT);
      }
  }
  

  
  Results in:
  Code size = 4368
  Total size = 4540
  
  But if I move those led variables to global:
  

  #include <PropWare/PropWare.h>
  #include <PropWare/gpio/pin.h>
  
  using PropWare::Pin;
  using PropWare::Port;
  
  const Pin led1(Port::P17, Pin::Dir::OUT);
  const Pin led2(Port::P16, Pin::Dir::OUT);
  
  int main () {
      led1.start_hardware_pwm(4);
  
      while (1) {
          led2.toggle();
          waitcnt(CLKFREQ / 4 + CNT);
      }
  }
  

  
  Code size = 4392
  Total size = 4588
  
  I get a larger code size. Now, I haven't looked through the generated assembly, but I would bet this isn't any less efficient. I bet it's just letting the compiler do more static init and less at runtime.
  
  
  Now, with that knowledge, I'll leave it to you to find an appropriate balance between knowing space requirements for your program and "globals are evil"