Database support for Spin?
bGatenby
Posts: 1
I'm a college student so please excuse my ignorance if this is a stupid question.
How might someone go about using a large database upwards of 8 gigs with an embedded system built upon the propeller? Is there a way to manage a database in spin for such a system?
The idea is to make a stand alone unit to process a large database which requires a large amount of computation to derive a value(s) within the database. The end goal will be to use multiple prop chips working together to process the database in a timely manner but the main problems I can foresee is managing the database, how to store said database, and possible bottlenecking at the storage interface. Any input is welcome, really I'm just evaluating options in an attempt to put together a plan in advance.
How might someone go about using a large database upwards of 8 gigs with an embedded system built upon the propeller? Is there a way to manage a database in spin for such a system?
The idea is to make a stand alone unit to process a large database which requires a large amount of computation to derive a value(s) within the database. The end goal will be to use multiple prop chips working together to process the database in a timely manner but the main problems I can foresee is managing the database, how to store said database, and possible bottlenecking at the storage interface. Any input is welcome, really I'm just evaluating options in an attempt to put together a plan in advance.
Comments
Leon
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Amateur radio callsign: G1HSM
Suzuki SV1000S motorcycle
Post Edited (Leon) : 6/23/2008 6:57:19 PM GMT
As far as memory available to the prop, 8 Gigs is enormous.
There has been some projects and discussions around IDE hard drives. You'd have to define you're own SQL (or other database language), storage schema, type system. You'd also be looking at a lot of external ram. I think you'll run short on pins, and have a huge programming task ahead of you. I don't think your gains (if there are any) will be worth the effort.
Looking for a capstone design project or ???
1) The current Propeller has only 32K of memory for Spin program and the program's data
2) There are only 32 I/O pins of which 4 are somewhat spoken for.
This limits the bandwidth possible for data transfers and inter-Propeller communication
Frankly, you'd be better off using a multicore embedded PC running Linux and one of the open-source database engines.
If this is for a project and you're required to do it with multiple Propellers, that's another situation. It's a major effort and the issues have little to do with the use of Spin and much more with the issues around constructing a multiprocessor database system with processors with limited memory and limited bandwidth to external (database) memory and for interprocessor communications.
If a client is possible, a serial interface/modem could be used to connect to a server.
Don't too excited about this [noparse]:)[/noparse] Seems like an overwhelming amount of work.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Post Edited (jazzed) : 6/23/2008 7:39:21 PM GMT
Another point is the SD card library is set up to recognize fat16 formatted cards. The maximum size partition for fat16 is 4GB (64k cluster size), and im not sure if there is a limitation of the cluster size in the driver. The upshot is you would have to either rewrite the driver to recognize fat32 formated cards (not an easy task), or use multiple cards (each would have a seperate CS pin) this would also require rewritting the driver, but it is a much more straight forward task.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Paul Baker
Propeller Applications Engineer
Parallax, Inc.
Graham
1. Use the DBF format i.e. it is simple and well documented
2. Database as file on SD card
3. Database can be created/edited etc using PC and then copied to the SD card.
For more info about the DBF structure:
www.dbase.com/knowledgebase/int/db7_file_fmt.htm
If anybody wants it I have a DBF reader that I coded in C# some years ago that could be ported to spin.
Cheers