SD Cache Loader Program

Dave Hein · 2012-11-04 12:34

I'm trying to write a program that does the same thing as sd_cache_loader, but runs as a user program. Basically, I want to prompt the user to enter his SD pin configuration, save it to EEPROM and then start up an XMMC program that's on the SD card. I've looked at the sd_cache_loader program in the sdloader directory, but I can't figure out exactly how to do this. Does anybody have any suggestion? Steve? David?

David Betz · 2012-11-04 18:15

Dave Hein wrote: »

I'm trying to write a program that does the same thing as sd_cache_loader, but runs as a user program. Basically, I want to prompt the user to enter his SD pin configuration, save it to EEPROM and then start up an XMMC program that's on the SD card. I've looked at the sd_cache_loader program in the sdloader directory, but I can't figure out exactly how to do this. Does anybody have any suggestion? Steve? David?

You mean any suggestions about how to understand the sd cache loader code? If you have any questions about how it works I'd be happy to answer them.

Dave Hein · 2012-11-05 05:09

Yes, how does it work? Do you have any documentation on it? I've built the cache loader, and then tried loading it with propeller-load, but it hangs on the first step when it tries to start up the cache driver. When I load it with propeller-load I don't see any indication that it's setting up the SD pins. Do I need to call the mount routine explicitly in this case? Would that even work? Does it use the same SD driver that we use with the FileDriver code, or is this a special SD driver that includes the cache driver, or is the cache driver completely seperate from the SD driver?

What is in the info structure? Do I need to populate that before I start the cache driver? Where does the 8K of cache reside in memory -- start, end or somewhere in the middle. Does the sd cache loader code take up any space after it starts up the XMMC program, or is it all available to the XMMC program except for the 8K cache?

That's all the questions I can think of right now. Thanks.

David Betz · 2012-11-05 06:56

Dave Hein wrote: »

Yes, how does it work? Do you have any documentation on it? I've built the cache loader, and then tried loading it with propeller-load, but it hangs on the first step when it tries to start up the cache driver. When I load it with propeller-load I don't see any indication that it's setting up the SD pins. Do I need to call the mount routine explicitly in this case? Would that even work? Does it use the same SD driver that we use with the FileDriver code, or is this a special SD driver that includes the cache driver, or is the cache driver completely seperate from the SD driver?

What is in the info structure? Do I need to populate that before I start the cache driver? Where does the 8K of cache reside in memory -- start, end or somewhere in the middle. Does the sd cache loader code take up any space after it starts up the XMMC program, or is it all available to the XMMC program except for the 8K cache?

That's all the questions I can think of right now. Thanks.

I'll answer your questions later after my work day is over but one thing that may be confusing you is that propeller-load patches these images before it downloads them to the Propeller. They certainly won't work if those patches are not done. I'll try to put together a document describing how the load process works to make this clearer. The main reason for the patching is to allow information from the board configuration files to be made available to the loader code on the Propeller.

Dave Hein · 2012-11-05 07:18

Yes, I understand a little bit about the patching of the SD pin information. What I want to do is to write a program that does what the SD cache loader does, but prompts the user for board-specific information. My reason for this is that I want to be able to distribute a program that consists of an AUTORUN.PEX program that resides on an SD card, and a boot program that resides on the EEPROM. These binaries would be used by people who don't used the PropGCC tools, and are only interested in running the binaries. They will have their own unique pin assignments for the SD card, and they will only have to configure the boot program the first time they run it. After that, it will boot up the PEX program without any user interaction.

David Betz · 2012-11-05 09:54

Dave Hein wrote: »

Yes, I understand a little bit about the patching of the SD pin information. What I want to do is to write a program that does what the SD cache loader does, but prompts the user for board-specific information. My reason for this is that I want to be able to distribute a program that consists of an AUTORUN.PEX program that resides on an SD card, and a boot program that resides on the EEPROM. These binaries would be used by people who don't used the PropGCC tools, and are only interested in running the binaries. They will have their own unique pin assignments for the SD card, and they will only have to configure the boot program the first time they run it. After that, it will boot up the PEX program without any user interaction.

That makes sense. I'll try to put together a description of the SD boot process later this evening or maybe tomorrow.

David Betz · 2012-11-05 09:58

Here is a paragraph I found from a loader internals document I started writing a while ago. Unfortunately, it doesn't seem like the sd loader will do what you want since it can't load LMM or CMM programs in its current form.

SD Loader
The SD loader (sd_loader.c) is a program that can either be loaded directly into hub memory or written to EEPROM to execute on Propeller reset. It mounts an SD card and loads the file ‘autorun.pex’ from the root directory of the card into external memory. It must be an XMM or XMMC program. Its purpose is to provide a way for a program that uses external memory to automatically start on reset.
The SD loader does not have the ability to load an LMM program since those programs can be written directly to EEPROM .
The SD loader uses a SPI SD card driver (sd_driver.spin) to handle SD card sector reads.

Dave Hein · 2012-11-05 10:05

David Betz wrote: »

Here is a paragraph I found from a loader internals document I started writing a while ago. Unfortunately, it doesn't seem like the sd loader will do what you want since it can't load LMM or CMM programs in its current form.

I'm not sure where the confusion is in the description of what I want to do. I don't want to load an LMM or CMM program from SD. I want to load an XMMC program from SD. Isn't that what the SD cache loader does? The problem is that I don't know what the user's configuration is, so I need a loader that boots from EEPROM that can query the user to enter his SD pin configuration.

David Betz · 2012-11-05 11:11

Dave Hein wrote: »

I'm not sure where the confusion is in the description of what I want to do. I don't want to load an LMM or CMM program from SD. I want to load an XMMC program from SD. Isn't that what the SD cache loader does? The problem is that I don't know what the user's configuration is, so I need a loader that boots from EEPROM that can query the user to enter his SD pin configuration.

Okay, if you want to load an xmmc program it should be fairly easy to do. As I said, I can look at this later or tomorrow. Unfortunately, I'm in the middle of my "day job" now. Sorry!

jazzed · 2012-11-05 17:37

Dave Hein wrote: »

... I want to load an XMMC program from SD. Isn't that what the SD cache loader does? The problem is that I don't know what the user's configuration is, so I need a loader that boots from EEPROM that can query the user to enter his SD pin configuration.

So, you want to run an XMMC program from SD card that is booted from a monitor program stored in EEPROM? SD XMMC is a little slow. There is a faster micro-SD plug-in alternative coming of course

Will SD card be mandantory?

Will the cache driver be stored on SD card too? What about other types of XMMC programs? Will you support that too? I'm guessing that one could put some of the simpler .cfg files on SD card and parse them to support all types.

Dave Hein · 2012-11-05 18:19

You may have seen the pfth ANS Forth interpreter that I posted in another thread. It's an XMMC program that loads and executes Forth files from an SD card. Since it requires an SD card I know that I will always be able to run it from the SD card. For my purposes I have been using SimpleIDE to compile and run the program. BTW, I really like the file transfer feature in SIDE. It makes it easy to write Forth code on the PC, transfer it to the Prop's SD card, and then run the Forth interpreter -- all from SIDE.

Most Forth developers don't use PropGCC, so I need a way to provide the cache loader and XMMC binaries, which can then be configured for their particular SD pin connections. My plan was to adapt David's SD cache loader to do this, but I can't quite figure out how to do this. I'm sure I can figure it out eventially, but I was hoping to save some time.

At some point I want to run XMMC programs under spinix also. In that case the cache loader program would be a file on the SD card. This could be more flexible and use .cfg files to allow execution from other types of memory, such as flash or RAM. However, the EEPROM boot program is my more immediate need, and the OS version of the loader could be done later.

jazzed · 2012-11-05 19:18

Dave Hein wrote: »

You may have seen the pfth ANS Forth interpreter that I posted in another thread. It's an XMMC program that loads and executes Forth files from an SD card.

Yes. I downloaded it today to try and understand the issue better. Looks like a nice piece of work so far.

Dave Hein wrote: »

Since it requires an SD card I know that I will always be able to run it from the SD card.

Makes sense.

Dave Hein wrote: »

BTW, I really like the file transfer feature in SIDE. It makes it easy to write Forth code on the PC, transfer it to the Prop's SD card, and then run the Forth interpreter -- all from SIDE.

Thanks. I figured people didn't need to move their sd cards around all the time risking breaking things. It uses a propeller-load feature written by David Betz.

Dave Hein wrote: »

Most Forth developers don't use PropGCC, so I need a way to provide the cache loader and XMMC binaries, which can then be configured for their particular SD pin connections.

It's a nice idea to not depend on the developer to make specials for everyone. GCC does that to a point now, and it looks like our cache driver design decisions make it even more important and useful in this case. It would be great if we can easily extend that idea further in a user feature.

Dave Hein wrote: »

My plan was to adapt David's SD cache loader to do this, but I can't quite figure out how to do this. I'm sure I can figure it out eventially, but I was hoping to save some time.

It looks like you can just stuff some of the values in the main program (propgcc/loader/src/sd_loader.c line 88). I could be wrong of course.

    params[2] = info->cache_param1;
    params[3] = info->cache_param2;

The meanings of these parameters are in various cache files and David's propeller-load document. I don't see the cache-param1/2 fields explained for sd card - presumably because they can take on various meanings. The parameters are in propgcc/loader/spin/sd_cache.spin.

' sdspi_config1: 0xiiooccpp - ii=mosi oo=miso cc=sck pp=protocol
' sdspi_config2: 0xaabbccdd - aa=cs-or-clr bb=inc-or-start cc=width dd=addr
' the protocol byte is a bit mask with the bits defined above
'   if CS_CLR_PIN_MASK ($01) is set, then byte aa contains the CS or C3-style CLR pin number
'   if INC_PIN_MASK ($02) is set, then byte bb contains the C3-style INC pin number
'   if MUX_START_BIT_MASK ($04) is set, then byte bb contains the starting bit number of the mux field
'   if MUX_WIDTH_MASK ($08) is set, then byte cc contains the width of the mux field
'   if ADDR_MASK ($10) is set, then byte dd contains either the C3-style address or the value to write to the mux field


' These two values get patched by the loader
params
sdspi_config1 long    $090a0b13 ' for the c3 (di=9, do=10, clk=11, protocol=CS_CLR_PIN_MASK|INC_PIN_MASK|ADDR_MASK)
sdspi_config2 long    $19080005 ' for the c3 (clr=25, inc=8, addr=5)

Dave Hein wrote: »

At some point I want to run XMMC programs under spinix also. In that case the cache loader program would be a file on the SD card. This could be more flexible and use .cfg files to allow execution from other types of memory, such as flash or RAM. However, the EEPROM boot program is my more immediate need, and the OS version of the loader could be done later.

Haven't looked at spinix lately. I always liked it. Do you have any other ideas for improvements? Maybe GCC programs can be used on it later for hardware abstraction.

David Betz · 2012-11-05 20:31

Okay, it looks like the SD loader is a bit complicated. Mostly, this is because it has to patch lots of stuff into the sd_loader.spin file before downloading it to the Propeller. It patches the following things:

1) Cache parameters (size, param1-4)
2) The load target. This determines whether the .pex file is loaded into RAM at 0x20000000 or flash at 0x30000000
3) The cache driver itself
4) SD SPI parameters
5) The SD card driver

Once it patches all of that into the sd_loader.spin binary image it downloads it either to hub memory or EEPROM.

When the sd_loader runs, it

1) loads the sd card driver and initializes the filesystem code
2) opens the autorun.pex file
3) loads and starts the XMM kernel from the .pex file (the kernel is idle at start)
4) loads the user program to either RAM or flash
5) stops the COG running the sd driver
6) chains to a secondary loader (vm_start)
7) the secondary loader initializes hub memory
8) the secondary loader tells the kernel COG to start and then stops itself

The use of the secondary loader allows the user program to have full access to all of hub memory except what is used by the cache driver.

David Betz · 2012-11-05 21:00

I was thinking about this a bit and it seems like one way that we might be able to achieve your goals would be to make a generic way of writing these patchable programs and also to have an option for propeller-load to just write out a .binary file with the patched program rather than downloading it to the Propeller. Or maybe it would be better to make this a separate program. I guess I would need to know what are the constants for your target boards and what are the variables. You've already said that the SD SPI pin settings need to be settable through a program that boots from EEPROM and presents the user with a configuration screen. What about the cache driver? Will you always be using the same cache driver or does that need to be configurable as well?

Rsadeika · 2012-11-06 03:44

At some point I want to run XMMC programs under spinix also.

Now, this seems like a worthwhile project, since you are wrapping up your ANS Forth academic exercise. I was also thinking that spinix is missing a way of using or creating something like a batch or script file. If I remember correctly, with that you can easily create "new" commands that could reside on the SD or in flash memory. With that, and the ability to use the SD, and flash memory, this could be a very useful system, but that is just my opinion.

Ray

Dave Hein · 2012-11-06 04:07

David Betz wrote: »

I was thinking about this a bit and it seems like one way that we might be able to achieve your goals would be to make a generic way of writing these patchable programs and also to have an option for propeller-load to just write out a .binary file with the patched program rather than downloading it to the Propeller. Or maybe it would be better to make this a separate program. I guess I would need to know what are the constants for your target boards and what are the variables. You've already said that the SD SPI pin settings need to be settable through a program that boots from EEPROM and presents the user with a configuration screen. What about the cache driver? Will you always be using the same cache driver or does that need to be configurable as well?

For my current applicaton I will always use the SD cache driver. Later on for spinix I would probably like to mimic the current loader operation and use a .cfg file to load a specific driver that support flash, RAM, etc.

So I have only been focusing on sd_cache_loader. I've ignored the sd_loader program. Do I need to run sd_loader before running sd_cache_loader?

David Betz · 2012-11-06 04:54

Dave Hein wrote: »

For my current applicaton I will always use the SD cache driver. Later on for spinix I would probably like to mimic the current loader operation and use a .cfg file to load a specific driver that support flash, RAM, etc.

So I have only been focusing on sd_cache_loader. I've ignored the sd_loader program. Do I need to run sd_loader before running sd_cache_loader?

Okay, sorry. I thought you were looking at the sd_loader program. I guess I didn't read your messages carefully enough. The sd_cache_loader is a bit different. It doesn't load the user code since it will be run directlly from the SD card. Instead, it loads a cluster map of the .pex file so that the sd cache driver doesn't need to understand the FAT filesystem. Other than that, it requires the same sort of patching from .cfg data as the SD loader.

No, you don't need the sd_loader if you're using the sd_cache_loader. They are mutually exclusive.

So what you want to do at this point is modify the sd_cache_loader so that instead of using SD SPI settings from a .cfg file, it instead puts up some sort of UI to allow the user to select these values? Will the UI use the serial connection to the Propeller or TV/VGA and keyboard?

Rsadeika · 2012-11-06 05:19

Will the UI use the serial connection to the Propeller or TV/VGA and keyboard?

This is a very interesting and important question to consider. Since I use items that do not have TV/VGA and keyboard access, I think that the serial connection would be the way to go, for the time being. I also think that VGA is more accessible than TV, unless of course there is a platform that has an HDMI port available?

Ray

Dave Hein · 2012-11-06 05:48

David Betz wrote: »

So what you want to do at this point is modify the sd_cache_loader so that instead of using SD SPI settings from a .cfg file, it instead puts up some sort of UI to allow the user to select these values? Will the UI use the serial connection to the Propeller or TV/VGA and keyboard?

Yes, I want to make a version of sd_cache_loader that prompts the user to enter the pin numbers used for the SD card, and then run an XMMC program on the SD card using the SD cache driver. The UI will use the serial connection.

What I have done so far is to run make in the propgcc/loader/sdloader directory. I then cd into the include directory and run "propeller-load -b c3 sd_cache_loader.elf -r -t". It doesn't get past the first step of "Initializing cache driver". I assume it's sitting in the "while (xmm_mbox[0]);" loop. I'm guessing that I need to patch a location with something -- maybe it needs the SD pin configuration at this point. I'll look at the Spin file that provides the SD driver to see if I can patch that.

EDIT: I'm guessing I need to initialize the SdLoaderInfo struct before starting up the cache driver. I assume cache_size is 8192. What should I put in cache_param1 - 4, and also the flags variable. Where do I put the SD pin configuration? Do I need to put the address of xmm_mbox in hub location $6? Are there any other hardwired addresses I need to set up?

David Betz · 2012-11-06 06:12

Hi Dave. I'm going to have to resume this discussion later tonight after work (while waiting for the election returns to come in!).

Dave Hein · 2012-11-06 06:46

Yes, I'm about to head out to vote, and then on to work myself. I'm hoping for the best in the election returns tonight. I don't want to see another $6 trillion added to the national debt during the next four years!

David Betz · 2012-11-07 07:48

Sorry, I intended to look at this last night but ended up, like many others probably, spending the entire night watching election returns. I'll try again tonight.

Dave Hein · 2012-11-07 07:59

I looked at it a bit more, and I suspect that propeller-load starts the SD cache driver cog before it loads sd_cache_loader.elf. It seems like what is needed is a version of propeller-load that runs on the Prop. Instead of downloading code from a computer to the Prop, it would boot from EEPROM on the Prop, and load programs from the SD card. Of course, this is tricky since all PropGCC programs run at the low end of memory. It would require small loader programs that would reside at high memory or in a cog.

David Betz · 2012-11-07 08:09

Dave Hein wrote: »

I looked at it a bit more, and I suspect that propeller-load starts the SD cache driver cog before it loads sd_cache_loader.elf. It seems like what is needed is a version of propeller-load that runs on the Prop. Instead of downloading code from a computer to the Prop, it would boot from EEPROM on the Prop, and load programs from the SD card. Of course, this is tricky since all PropGCC programs run at the low end of memory. It would require small loader programs that would reside at high memory or in a cog.

I don't think this is true. The SD filesystem code used by sd_cache_loader.c uses the sd_cache driver for low-level SD card access. That driver is included in the image that is either loaded into hub memory or written to eeprom. This pretty much has to be the case since the idea is to be able to start a program off of an SD card at reset without having a PC attached at all. I'll verify this later but I'm pretty sure this is how it works. This means, of course, that you can't access or even initialize the filesystem code until after you've collected the SD pin information from your user. The UI to handle that will have to be added before the call to load the cache driver that is at the very start of sd_cache_loader.c.

David Betz · 2012-11-07 08:24

I have another question about what you're trying to do. How do you want to handle parameters like clock settings, serial baud rate, or even serial pins? Can we assume those are always fixed?

Dave Hein · 2012-11-07 08:25

Yes, I tried that, but it didn't get past trying to initialize the cache driver. I really just need to gather all the pieces of code that are used and add a few debug prints to see how it works. After all, all the information on how it works is contained in the self-documenting code.

David Betz · 2012-11-07 08:31

Dave Hein wrote: »

Yes, I tried that, but it didn't get past trying to initialize the cache driver. I really just need to gather all the pieces of code that are used and add a few debug prints to see how it works. After all, all the information on how it works is contained in the self-documenting code.

Did you patch the sd cache driver with the selected SPI pins before loading it? Anyway, I'll try this tonight and I'll write a description of what is necessary.

Here is what I will attempt:

1) make propeller-load able to write out a .binary file already patched with .cfg file entries for a specified board.
2) add code to sd_cache_loader.c to use the serial port to prompt for SD pin configuration using the defaults from the selected board configuration.
3) patch the sd cache driver with the selected SPI pins.
4) start the cache driver
5) mount the filesystem
6) setup to run the file "autorun.pex" using the sd cache
7) load the kernel and the user program and start it

Is that what you want? What about the name of the program to run? Should I also prompt for that?

Dave Hein · 2012-11-07 10:27

That's basically what I need, except for item 1. My goal is to provide two binary files to the user. One of them is the PEX file that they copy to their SD card, and the other is a modified sd_cache_loader.binary file that they write to their EEPROM using the Prop Tool or BST. When it boots up they will be prompted for the SD pin numbers. autorun.pex is OK for now, or I might change it to pfth.pex, but I can change that later on. I don't need a default configuration, but I can also add that later if needed. Thanks.

David Betz · 2012-11-07 10:34

Dave Hein wrote: »

That's basically what I need, except for item 1. My goal is to provide two binary files to the user. One of them is the PEX file that they copy to their SD card, and the other is a modified sd_cache_loader.binary file that they write to their EEPROM using the Prop Tool or BST. When it boots up they will be prompted for the SD pin numbers. autorun.pex is OK for now, or I might change it to pfth.pex, but I can change that later on. I don't need a default configuration, but I can also add that later if needed. Thanks.

Step 1 is useful if you don't want to have to duplicate what propeller-load already does reading information from .cfg files and using it to patch .binary files.

David Betz · 2012-11-07 17:29

I modified propeller-load so that it writes out a sd_cache_loader.binary file if given the options -z and -s at the same time. This writes a binary file with the sd_cache driver patched into it as well as the normal stuff that always gets patched like the clock settings. It turns out that the sd_cache_loader doesn't use the cache parameters even though it has a parameter block for them. So, I typed the following command:

propeller-load -b c3 -z -s

This wrote a file called sd_cache_loader.binary which I was able to load into a C3 and it successfully started the file autorun.pex on the C3's SD card.

Of course, this isn't really what you want because it doesn't prompt for SD pins when it starts. I'm mulling over how to handle this. The problem as you've already noticed is that you can't just load sd_cache_loader.elf because it doesn't have a cache driver patched into it and hence won't work. Either the sd cache driver needs to be linked in directly rather than being patched or propeller-load or some other tool needs to be able to patch the cache driver into the modified verison of sd_cache_loader.

The problem is, this idea of just linking in the cache driver violates the model used by propeller-load where board-specific stuff is not included in the .elf files but instead merged at load time from a board configuration file. I'm not sure how to handle this yet. I'll think about it...

Dave Hein · 2012-11-07 18:45

So the the sd_cache_loader.binary file is the same thing as the image you write into the EEPROM when just specifying the -z flag, correct? The problem is that I don't want to specify the board's SD pins at the time the binary is written, but they are specified when the binary is run.

It seems like what is needed is a program that includes the cache driver DAT file, and knows where the SD pin descriptor needs to be patched into the DAT file. This should be a predicable location since I am only interested in the SD cache driver. After the program prompts the user for the SD pins, it then patches the DAT file, and then starts the SD cog. It then should be able to run the rest of the sd_cache_loader program at that point, correct?

SD Cache Loader Program

Comments