Good news, Bad news on my own filesystem project
localroger
Posts: 3,452
I have written and discarded about 100K of code in the last couple of weeks, and that's even with the phone ringing and a few unscheduled trips to remote locations to save peoples' bacon.· I now have 90% of a filesystem which I am about to throw away because I am going to use it to build something so awesome you would need sunglasses to read the release announcement.
The bad news is that when I showed my boss what I was up to, he forbade me to release it publicly, because it was such an advance and it would give our competitors too much of a head start if they ever try to duplicate our ultimate product.· I was thinking that the filesystem would be a relatively small part of that, but he might have a point with this new idea.
But you can't copyright an IDEA.· So how 'bout an idea?
My block driver will be built upon sdspifasm_ls(2), the version hacked to enable reading sdhc SD cards.· That will take a cog.
My main driver will be a stream driver, written in Spin and living in a cog.· It might be cognew'ed to live on its own via STANDALONE or share a cog with other functions, such as maintaining a user interface, by other code that calls START then SERVICE with sufficient frequency.
The stream driver will have all the I/O objects under it -- text, keyboard, serial, and of course the SD block driver.· They will be accessed the same way as open files, via requests with standardized file handles.· (io.putstr(io#stdout, string("HELLO WORLD"))
The stream access object will be small, leaving messages for the stream driver in the usual PASM access manner.· It will have a limited VAR footprint so having multiple obj's have it under them will be no problem.· Most importantly, this will be a multi-cog implementation -- processes in any cog will be able to request I/O services to any stream, which can include open SD files.
The block driver will allow multiple simultaneous open files, with dynamic allocation of a random number of 512 byte segment buffers.· (The number of those can be as low as 1 if memory is tight, but performance will of course nosedive.)· Important file information will be separately buffered in an open-file structure much smaller than a segment.· Despite the code to do this the memory savings are huge, and FILES=10 will have the effect of BUFFERS+=1.
This will not be a high-performance filesystem in the bandwidth sense; in some ways it is exactly the opposite of what rokicki and lonesock have been aiming for.· It's targeted to an app that might have several separate streams logging data at a modest rate to different files, a user interface looking at that data and altering config files, and a web interface serving up views and maybe making mods.· At modest data rates, but all at the same time.
I will be adding sdspifemto-style SPIN support to sdspiqasm_ls(2).· With the added caveat that I will support noncontiguous files, by allowing the spin driver to load a LIST of segments built by Spin code that examines the FAT, instead of a contiguous area of the SD card.
In particular, I have become convinced that there are some dead-end things about the way most obex objects are implemented.· They're very convenient one at a time, but why must the text, serial, and SD object ALL have str, hex ,dec, bin, etc. methods?· And why did nobody think of doing a fixed-width dec method (hint:· you can implement a string buffer by declaring a short array of longs in the local vars.· Even if you want bytes, you can use BYTE@ to use the reserved memory.)
I have most of this working already in single-cog style.· That version, which I won't ever finish now, has a few bugs which mess up the FAT for reasons I now won't ever bother to debug, but it works spectacularly well for things like wildcard rename and dir.· (True wildcards with * and ? interpreted like DOS does.)· So far it's just FAT16 (I'm using sdspifemto for the block driver) but it handles multicluster directories just fine.
So my message is ... think streams.· And pull all that hex, bin, dec Smile out of the low-level drivers and put it in a higher level where it belongs.
Meanwhile, I have a mostly functional version of PropCMD that just got orphaned.· After stripping it for what works I will probably never look at it again.· Such is the nature of R&D.
The bad news is that when I showed my boss what I was up to, he forbade me to release it publicly, because it was such an advance and it would give our competitors too much of a head start if they ever try to duplicate our ultimate product.· I was thinking that the filesystem would be a relatively small part of that, but he might have a point with this new idea.
But you can't copyright an IDEA.· So how 'bout an idea?
My block driver will be built upon sdspifasm_ls(2), the version hacked to enable reading sdhc SD cards.· That will take a cog.
My main driver will be a stream driver, written in Spin and living in a cog.· It might be cognew'ed to live on its own via STANDALONE or share a cog with other functions, such as maintaining a user interface, by other code that calls START then SERVICE with sufficient frequency.
The stream driver will have all the I/O objects under it -- text, keyboard, serial, and of course the SD block driver.· They will be accessed the same way as open files, via requests with standardized file handles.· (io.putstr(io#stdout, string("HELLO WORLD"))
The stream access object will be small, leaving messages for the stream driver in the usual PASM access manner.· It will have a limited VAR footprint so having multiple obj's have it under them will be no problem.· Most importantly, this will be a multi-cog implementation -- processes in any cog will be able to request I/O services to any stream, which can include open SD files.
The block driver will allow multiple simultaneous open files, with dynamic allocation of a random number of 512 byte segment buffers.· (The number of those can be as low as 1 if memory is tight, but performance will of course nosedive.)· Important file information will be separately buffered in an open-file structure much smaller than a segment.· Despite the code to do this the memory savings are huge, and FILES=10 will have the effect of BUFFERS+=1.
This will not be a high-performance filesystem in the bandwidth sense; in some ways it is exactly the opposite of what rokicki and lonesock have been aiming for.· It's targeted to an app that might have several separate streams logging data at a modest rate to different files, a user interface looking at that data and altering config files, and a web interface serving up views and maybe making mods.· At modest data rates, but all at the same time.
I will be adding sdspifemto-style SPIN support to sdspiqasm_ls(2).· With the added caveat that I will support noncontiguous files, by allowing the spin driver to load a LIST of segments built by Spin code that examines the FAT, instead of a contiguous area of the SD card.
In particular, I have become convinced that there are some dead-end things about the way most obex objects are implemented.· They're very convenient one at a time, but why must the text, serial, and SD object ALL have str, hex ,dec, bin, etc. methods?· And why did nobody think of doing a fixed-width dec method (hint:· you can implement a string buffer by declaring a short array of longs in the local vars.· Even if you want bytes, you can use BYTE@ to use the reserved memory.)
I have most of this working already in single-cog style.· That version, which I won't ever finish now, has a few bugs which mess up the FAT for reasons I now won't ever bother to debug, but it works spectacularly well for things like wildcard rename and dir.· (True wildcards with * and ? interpreted like DOS does.)· So far it's just FAT16 (I'm using sdspifemto for the block driver) but it handles multicluster directories just fine.
So my message is ... think streams.· And pull all that hex, bin, dec Smile out of the low-level drivers and put it in a higher level where it belongs.
Meanwhile, I have a mostly functional version of PropCMD that just got orphaned.· After stripping it for what works I will probably never look at it again.· Such is the nature of R&D.
Comments
One person's Smile is another person's jewels.
I don't know about your file system proposal (not enough experience here) but I agree with you about the obex - although I see this as a reflection of SPIN itself. SPIN objects (however well crafted they are) are very 'concrete' objects and are difficult to use as part of a higher level construct - especially an essentially 'abstract' construct which is itself intended to be used only to construct other things.
On the other hand, I keep having to remind myself that the Propeller is just a microcontroller, that it only has 32kb RAM (or even worse, only 2kb per cog) - and yet what we're all expecting it to do (or actually managing to get it to do) is just absolutely mind-boggling
Ross.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Catalina - a FREE C compiler for the Propeller - see Catalina
But I had to smile when you mentioned your boss doesn't want to give any other competitor a "head start". Trouble with copying other people's technology though is that they never really understand it and will always be second-rate, so I don't worry "too" much (just a little).
As to multiple files and dynamic buffers etc I probably have done similar things on other processors and when I get a "round-tuit" I may do the same on the Prop unless somebody else saves me the trouble (memory memory, we need more memory). There is so much high-quality stuff being done that we all benefit from that it seems we are almost under obligation to contribute also. Your boss should acknowledge this or at least perhaps make a generous payment to the authors of various objects that he would otherwise have had to pay for many times over (just kidding, but it would be proper considering).
*Peter*
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
--Steve
Propalyzer: Propeller PC Logic Analyzer
http://forums.parallax.com/showthread.php?p=788230
But that's just a buzz-word for "character-oriented"!
Under the hood, something like printf is always stream oriented, even on block-oriented devices. Handling the blocks is something the device-driver does.
The concept of file-handles for the higher-level routines (like printf, hex, bin, dec, whatever) is what allows to write libraries at that level. And all device-drivers have to implement something like putch and getch and writeblock and readblock. The later two don't have to know how big such a block of the device is. That's all handled by the device driver.
Pointers to functions in SPIN? I don't know! But I also don't care.
Nick
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Never use force, just go for a bigger hammer!
The DIY Digital-Readout for mills, lathes etc.:
YADRO
Getting rid of duplicate formatting routines (or any duplicate functionality) is a good idea in terms of saving space in the Prop and only having to debug them once. Provided that the overhead of doing so in terms of space and time does not eat the gains hoped for.
No idea what Bill Henning is up to with Largos but he does talk in terms of message passing which sounds like it's heading in your "streams" direction well.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
Basically I need to get Morpheus to the point where I can start advertising it and increasing sales so I will have the time (hopefully supported by an income stream from Morpheus) do work on LAS and Largos a LOT. I also need to get a nice text mode driver running on Morpheus so I can have a great local console, and not have to run a serial terminal on a PC to interact with Largos. Ross will be targeting a Catalina kernel to Morpheus's XMM, so there is a decent chance of having LAS running right on Morpheus soon, not to mention the possibility of also running Catalina on it later
I should be getting back to Largos in about two weeks, and yes, Largos I/O can be viewed as if it was streams mapped over message passing. The system calls will seem very familiar (with a few minor changes).
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com - my site 6.250MHz custom Crystals for running Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
Morpheus & Mem+ Advanced dual Propeller SBC with XMM and 256 Color VGA
Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full
Nick, you know, " character-oriented " is in turn, just a buzz-word for pipes, which is·buzz, for·buffers, which is ...·shared memory
With minor technical distinction, " streams " is·more mnemonic than function.·
V.Gruesse!
·Howard
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBladeProp, RamBlade, TwinBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: Micros eg Altair, and Terminals eg VT100 (Index) ZiCog (Z80), MoCog (6809)
· Search the Propeller forums (via Google)
My cruising website is: ·www.bluemagic.biz·· MultiBladeProp is: www.bluemagic.biz/cluso.htm
Thats excellent!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.mikronauts.com - my site 6.250MHz custom Crystals for running Propellers at 100MHz
Las - Large model assembler for the Propeller Largos - a feature full nano operating system for the Propeller
Morpheus & Mem+ Advanced dual Propeller SBC with XMM and 256 Color VGA
Please use mikronauts _at_ gmail _dot_ com to contact me off-forum, my PM is almost totally full