v2.0.2 of Memory Storage Management Released
Bobb Fwed
Posts: 1,119
This is the largest single update since release. It was originally going to be a minor update and then a major, but a lingering bug held off the minor update.
Download the heavyweight and welterweight versions here: http://obex.parallax.com/objects/493/
Here are the key updates (from 1.3.3 to 2.0.0):
Fixed a bug that would make values become unset if name hashes collided and the first values set were renamed. Because of this bug, I highly encourage everyone who uses this object to update immediately.
Deleting no longer means wasting space. The software now "reserves" the name and data space used by deleted values, so they can be used when a new item of the same type is created.
Hash algorithm has been optimized in both speed and results. Reducing the likelihood of collision between consecutive naming. The new hash also allows for non-power-of-two table sizes (for added flexibility in working around existing code).
Added methods to allow arrays to be larger than the previous 255 byte limit (strings can be stored as arrays to allow for longer than 255 byte strings).
Changed some method names for terminology and logical reasons.
Optimizations in many of the methods.
Changes in 2.0.1:
Added a method to retrieve names with a wild card.
Fixed a couple minor name listing bugs.
Added a method to allow retrieving just specific parts of a stored array, so there is no need to cycle through the entire array to get one small portion of it.
And a couple other minor changes.
Changes in 2.0.2:
Name list can now be sorted alphabetically.
Added a method to get values of a specified type.
Post any questions, comments, suggestions, feature requests, or bugs on here. There were quite a few changes, and I think I've tested everything pretty thoroughly, but please post anything that does not function as one would expect, and I will look into it.
Download the heavyweight and welterweight versions here: http://obex.parallax.com/objects/493/
Here are the key updates (from 1.3.3 to 2.0.0):
Fixed a bug that would make values become unset if name hashes collided and the first values set were renamed. Because of this bug, I highly encourage everyone who uses this object to update immediately.
Deleting no longer means wasting space. The software now "reserves" the name and data space used by deleted values, so they can be used when a new item of the same type is created.
Hash algorithm has been optimized in both speed and results. Reducing the likelihood of collision between consecutive naming. The new hash also allows for non-power-of-two table sizes (for added flexibility in working around existing code).
Added methods to allow arrays to be larger than the previous 255 byte limit (strings can be stored as arrays to allow for longer than 255 byte strings).
Changed some method names for terminology and logical reasons.
Optimizations in many of the methods.
Changes in 2.0.1:
Added a method to retrieve names with a wild card.
Fixed a couple minor name listing bugs.
Added a method to allow retrieving just specific parts of a stored array, so there is no need to cycle through the entire array to get one small portion of it.
And a couple other minor changes.
Changes in 2.0.2:
Name list can now be sorted alphabetically.
Added a method to get values of a specified type.
Post any questions, comments, suggestions, feature requests, or bugs on here. There were quite a few changes, and I think I've tested everything pretty thoroughly, but please post anything that does not function as one would expect, and I will look into it.
Comments
Just a minor note, in your "_README_.txt" file, you mentioned "memory_demo2.spin" but actual file is "memory_demo.spin"
I don't foresee any more updates to this object for a while (other than catching the lightweight version up to 2.0 and of course). I had to get the deletion feature in for a project I am working on, and I discovered the rename bug while working on that. So now I have both of those out of the way, I will be working on other things, like my 10Mb prop-to-prop communication object (possibly getting it to 20Mb!) -- either way, there are features I need to add (for this same project) and some functionality anomalies that need to be fixed -- I've been getting requests for this anyway.
Please, let me know what you think of this object (and the added features).
I'll be studying & testing your v2.0.0 over the weekend as my current project badly need such features. Also, I've already starting working on multi-prop (got 2 to talk but hope to increase the bandwidth) so I'll definitely look into your next object next month.
I've been reading & testing out your v2.0.0. I seems to get a hang of it & its real helpful. I have a request if it doesn't sounds silly.
In MySQL, I usually could perform a query & store that to ram. E.g. select field from table where field like "LED_*". I'm thinking of using 1 EEPROM's upper 32k for storing all mounted prop's setup & pin assignments. I know I could write a query function to do that but thought you might have a better way of doing this. By doing this, I could query all "LED_*" pin assignment & send them over to LED module & query all "Servo_*" pin to Servo module in another prop etc.
I've tried with output from PST:
Added: Hmm ... I think maybe I could use the create_array instead.
A wild card at the end will be fairly fast, a wild card at the beginning will be a bit slower, two wild cards (beginning and end) will be quite slow. And I don't really see being able to do more than that (no internal wild cards). I can write it up and see how it goes.
Yes, storing the values as an array would be faster to store and access, and use less memory (because there is only one name being stored).
Thanks for the input! Let me know any other ideas or problems.
I'll work on getting an unlimited wild card system (as it should be).
Let me know if that functions as you'd expect.
Cheers!!
Added get_pattern_name_list, which returns names based on a wild card (*) pattern. RC1 and newer allows unrestricted wild card use.
Fixed bug where if you had deleted items and called get_full_name_list then get_name_list, items were in the name table twice
Changed set_name_pointer so it doesn't call get_name_list, it just uses the last get_*name_list returned value as the maximum value (faster and more functional with the new get_*name_list method)..
Added set_array_pointer to allow retrieving just specific parts of an array, without having to cycle through or read the entire array.
Fixed overrun behavior in get_*_parts_array that occurred if you tried to access more elements than existed in an array.
I forgot to mention, I have released version 1.3.4 of the lightweight branch. Download it here: http://obex.parallax.com/objects/671/
The issue I'm having is that the counter seems to skip forward to the next binary power once it reaches a certain number! So for instance, it will count 30,31,128,129,130...
I set up a separate program to test this, which consists of an initialization of the names and then a loop that increments the counter. Disregard some of the extra comments and odd code in there; they are put in by 12blocks.
pub _pub0'*(32,73):cognew($0,@_s$0) {Resets all of the permanent counter values to zero.}'*:{$} _eeprom.check_edit_create_long(string("count"),500)'*:_eeprom.check_edit_create_long($,$) _eeprom.check_edit_create_long(string("dry"),0)'*:_eeprom.check_edit_create_long($,$) _eeprom.check_edit_create_long(string("wet"),0)'*:_eeprom.check_edit_create_long($,$) _eeprom.check_edit_create_long(string("right"),0)'*:_eeprom.check_edit_create_long($,$) _eeprom.check_edit_create_long(string("darkness"),0)'*:_eeprom.check_edit_create_long($,$) {loops "moisture" and "light" readings so calibration of the potentiometer can be done.}'*:{$} _adc.start(13,12,11,10,2,2,12,1)'*:_adc.start($,$,$,$,$,$,$,$) repeat'*:repeat moisture:=_adc.getval(1)'* 1:$:=$ light:=_adc.getval(0)'*:$:=$ count:=_eeprom.get_dec(string("count"))'*:$:=$ _bsl.pause(500)'*:_bsl.pause($) _eeprom.check_edit_create_long(string("count"),count+1)'*:_eeprom.check_edit_create_long($,$) _bsl.pause(500)'*:_bsl.pause($)I'm watching the count value on the terminal. The "jump point" also seems to depend on the values I plug into the object, I can make it jump at either 32 or 64. And yeah, my EEPROM chip is 64K so I've got lots of room to store 5 or 6 longs. I can write any number I want with no issue. I just can't figure out why it isn't incrementing properly.
So what's going on here? Any ideas? Perhaps some weird interaction between 12blocks and this object?
Using some command line code from SD2.0 to make your EEPROM DB looks like the ancient DBASE III+, your get_pattern_name_list() works GREAT!!!
Thank you!!
Any final bugs or [small] feature requests? If not, I will update the welterweight branch and release version 2.0.1 on the obex.
Thanks a lot!
Ok, good to know. I'll do a comparison between my custom board and my propeller dev board to see if there's a difference.
Changes in 2.0.1:
Added a method to retrieve names with a wild card.
Fixed a couple minor name listing bugs.
Added a method to allow retrieving just specific parts of a stored array, so there is no need to cycle through the entire array to get one small portion of it.
And a couple other minor changes.
@MacTuxLin: I didn't get any hits on pre-written sorting code, so I will have to write and test it, which will take a while since it's not overly easy sorting strings that only addresses can be stored in an array. There is always room for it in v2.0.2.
So, v2.0.1 is working out great for me for now. I'm actually planning on writing up a mini-dBase-III+-like interface for accessing the EEPROM using your object partly for organizing my boards & also allowing our young users to tinker with simple database. Would it be ok if I know how I could send you the "wish-list" for your next versions?
Or just PM me.
With the basic I2C driver (SPIN), running my testing demo (only 16 names), it takes over 39 million cycles just to sort (almost a half second at 80MHz).
With the pasm I2C driver, running the same demo, it takes only 7.7 million cycles to sort.
This massive difference is because the sort method must read through each name from the EEPROM a number of times. The only real way to speed this up would be to place the entire name space into RAM, but that will likely use up a large amount of RAM, though it really depends on how the program is setup in the final configuration. The default values would require 256 longs (1024 bytes).
What are your thoughts? Is this too slow to use? Does anyone even use the PASM I2C (it sucks to burn a cog for EEPROM control)? How much name space do you use? Is it an option to place the entire name space into RAM?
It might sound silly but may I know if it would be faster (or make sense) to load the PASM I2C at will? Meaning, if I need to sort, I'll load this to a cog, sort, then cogstop?
Thanks a lot!
Added: I've tested adding 120, which is what I'll need, and it worked out great enough for me.
And it doesn't appear that just adding a cogid/cogstop to the end of the PASM stopfunc routine, and removing the loop in SPIN does the trick. For some reason, with this setup, it doesn't sort at all. My guess is that it has something to do with the SPIN I2C. I issue the stop command, but that looks like the pins are being held high, even if they are set as input. Some conflict with the two cogs and the IOs.
I have been using insertion sorting, but after some testing, I have found that shell sorting may be faster. With integers, it is way faster, but with strings, it is [sometimes] only marginally faster. Though the testing was done without any EEPROM access, so it is possible that insertion sorting still reads from EEPROM less (or something). But I will have to test it to find out.
It's amazing what a few hours of testing five different sorting methods will produce.
I have two sorting methods and I am trying to figure out which to use. One is insertion sort, the second is shell sort.
The relative speed of the two changes as the number of elements to sorts changes. Insertion sort is fast for small lists, shell sort is faster for long lists.
Insertion is 40% faster than Shell at 17 entries. (0.46s vs. 0.76s)
Insertion is 28% faster than Shell at 34 entries. (1.52s vs. 2.11s)
Insertion is 7% slower than Shell at 60 entries. (4.26s vs. 3.96s)
Insertion is 32% slower than Shell at 120 entries. (14.08s vs. 10.60s)
Insertion goes up approximately 3.3 times every time the entry count doubles.
Shell goes up approximately 2.7 times every time the entry count doubles.
So the question is: which would you prefer? Either slightly slower when the times are short, or faster when the times are long? Either way, over a couple seconds seems like ages and maybe not practical on this microcontroller. With the PASM I2C the speeds are about 5 times faster.
Yeap, I know, crazy week for me too. OK, I'm not sure about the rest of the members but a couple of seconds might seems a bit obvious at my end, though. Or how about this:
1. select LED_GroupA*
shows up 10 entries & just sort these 10?
So at any one time, the number of entries to sort is less than 20? Would this work?