It is very likely you are missing a point which is (considering how long it takes to set up an access on your board), performance of applications on your board using a cache would most likely increase by more than 50%.
Hi Jazzed,
Do you have any actual statistics on this yet?
The reason I ask is that it looks to me very much like I will not be able to fit the additional code required for handling Flash RAM into the current Catalina XMM Kernel - so it may be necessary to move the XMM access code out to another cog. If I have to do this, I would probably adopt your caching driver API, since I believe caching the serial RAM/Flash access may also result in a substantial performance boost - but this is mainly because serial access is many, many times slower than parallel access. On a board with reasonably fast parallel access to the SRAM, some (perhaps all!) of the benefit of caching will be lost due to the the additional synchronization required to coordinate activities between the kernel and the caching driver.
If you have any actual data on this, I'd be really interested to see it.
Advantages:
Cache abstracts the hardware type. Just replace the object.
I think that is the key - and if you have code already working then problem solved. It is now abstracted to a simple command to read and one to write.
The C3 probably is the hardware that BigSpin is going to run on for most people, so this is great. And for any different hardware, like you say, replace the object.
So ok, you can read and write longs to a flat memory space.
It seems to me you would not want to restrict that to the memory size of the C3. Nor even to 512k or even bigger memory chips. Given the prop is a 32 bit micro, it seems logical to choose a 32 bit memory space. Do that for the simulator, and use that for the code as well, and if you tell the compiler what actual memory you have, it can tell you the % you have used as you add to the code.
Dave seems to be moving ahead very rapidly with this. I'm intrigued with the the possibility that existing compilers could do most of the work, and the only 'new' code might be a linker.
Ross, synchronization in my API is only required when the data is not in the "current cache line."
You don't have to synchronize for every access, if that were required I would agree with you on lost benefit.
I'm too busy at this point to create code that would take approximately 4us to deliver one byte of data when 32 times that data can be delivered in only 3 times the wait. I might do it later as an academic exercise.
I don't have a Drac board to do such an experiment, but I know others do ;-)
Caching with BpgSpin is very likely to greatly improve speed. If you have a PUB and you request the first long from the cache code, and it delivers you n more longs, 32 or whatever, it is very likely you are going to need those longs.
The cache sounds like the answer to the memory problem. Code is already written and it works. Kudos to jazzed.
Going back to something Dave said
As an example, let's say a program consist of a top object and a single child object. This is split into two programs. The first program consists of the top object and a stubbed version of the child object. The stubbed version of the child object contains only the PUB and PRI statements from the child object, plus code that will performs a 32-bit jump into the correct method of the child program. The linker would just copy the parent program and the child program together, and it will add a 32-bit pointer that will be used by the child object stubs to call the methods in the child program. It sounds complicated, but it's actually pretty straight forward to implement.
This seems very logical. Does the same apply with some code you see on the obex where there are child processes with their own child processes? Not all code is two or more layers deep but some is.
More likely though, code in an obex is a group of pasm and spin, and if you can abstract that to a block, with pointers to go in and out of that block, it ought to decrease the modifications needed to obex code. Fantastic if indeed this is easy to implement, because this could be the last piece of the jigsaw.
The reason I ask is that it looks to me very much like I will not be able to fit the additional code required for handling Flash RAM into the current Catalina XMM Kernel - so it may be necessary to move the XMM access code out to another cog. If I have to do this, I would probably adopt your caching driver API ....
Ross, I forgot to mention that David Betz has already done this for C3 for you to use and his code should be easily integrated. If you add this, Catalina will automatically support SDRAM solutions as well. Zog may get jealous.
Ross, I forgot to mention that David Betz has already done this for C3 for you to use and his code should be easily integrated. If you add this, Catalina will automatically support SDRAM solutions as well. Zog may get jealous.
Hi Jazzed,
Yes, I know about David's work for the C3 - so I guess I only have to wait to find out the answer to my question. But I was thinking more generally about whether it would be a good idea to adopt a caching solution across all platforms - but my concern is that on some platforms this may actually be detrimental.
When I get some time, I'll do some experiments of my own.
As an example, let's say a program consist of a top object and a single child object. This is split into two programs. The first program consists of the top object and a stubbed version of the child object. The stubbed version of the child object contains only the PUB and PRI statements from the child object, plus code that will performs a 32-bit jump into the correct method of the child program. The linker would just copy the parent program and the child program together, and it will add a 32-bit pointer that will be used by the child object stubs to call the methods in the child program. It sounds complicated,
This seems very logical. Does the same apply with some code you see on the obex where there are child processes with their own child processes? Not all code is two or more layers deep but some is.
More likely though, code in an obex is a group of pasm and spin, and if you can abstract that to a block, with pointers to go in and out of that block, it ought to decrease the modifications needed to obex code. Fantastic if indeed this is easy to implement, because this could be the last piece of the jigsaw.
The approach I'm proposing could be used on OBEX code as well. I thought about it a bit more and realized that the linker could remove the stub routines and just concatenate all the objects together. Of course, the method tables would have to be fixed up to work correctly. I think we could still use the 16-bit entries up to a memory size of 256K. Beyond that, the method table entries would need to be increased to more than 16 bits.
but my concern is that on some platforms this may actually be detrimental.
It may. The point is, RamBlade will take less time to fetch a byte than a the cache algorithm can calculate the need to swap the current line. It would take twice as long to fetch a long than a cache calculation however. RamBlade can fetch one byte in 200ns (at 80MHz). It would take 1000ns to fetch one long with djnz overhead.
The C3 Cache COG uses the VM swap algoritm which has larger management code, but is more efficient in determining swap requirements because of cache line aging.
Ummm... I think you're giving me a little more credit than is due! :-)
Actually, the C3 cache code as it currently stands is really two separate caches. One is used to cache accesses to the 1mb SPI flash chip and the other is used to cache access to the 64k of SPI SRAM. Both caches are direct mapped caches. Since the GCC linker script places code in flash and data in SRAM this could be considered separate instruction and data caches. I have considered doing a unified set associative cache but haven't had time to work out an implementation.
Also, the main reason that the code in the C3 cache COG is bigger than the code in the SDRAM cache COG is that it includes code to erase and write the flash and also code for SD card sector I/O. This is to allow all of the C3 SPI devices (minus the A/D chip) to be managed by a single COG.
The approach I'm proposing could be used on OBEX code as well. I thought about it a bit more and realized that the linker could remove the stub routines and just concatenate all the objects together.
Can you explain that a bit more? Ok, my simplistic thinking it that you have something in the obex. It has a whole lot of spin PUBs and it might have some pasm code. If you hit 'compile' that obex code ought to compile to something, even if it can't run. The dat and the var data I think ends up concatenated and at the beginning. Then there is the PUB code.
So my understanding is you do something clever to that code. Do you create a list of calls to PUBs and group all those together so there is a list where everything is in order? Or in more general terms, how do you know where the start of each PUB has ended up?
I imagine if you do create some self contained code, with known locations for each PUBs and the code is internally consistent then you can use that in some way by a higher level 'main' program.
Ummm... I think you're giving me a little more credit than is due! :-)
Also, the main reason that the code in the C3 cache COG is bigger than the code in the SDRAM cache COG is that it includes code to erase and write the flash and also code for SD card sector I/O. This is to allow all of the C3 SPI devices (minus the A/D chip) to be managed by a single COG.
Nice! I've just spent a day battling to synchronize SD card access with XMM RAM access on the C3, since I currently do it from different cogs. This has advantages in that it doesn't use an extra cog when (for example) a program needs XMM access but doesn't need SD access - but it sure is a pain!
If I end up having to sacrifice a cog anyway to manage the SPI Flash, I may adopt your solution.
A Spin object consists of a method table, DAT section and PUB/PRI code. A binary file contains a 16-byte header that is used to set up the clock and the various pointers used by the spin interpreter. The VAR sections for all of the objects follow immediately after the last PUB/PRI method in the last object, and the stack begins after that. PASM code is treated as regular DAT data, and Spin doesn't really know the difference between PASM code and any other DAT data.
The linker just has to concatenate all of the objects together and fix up the pointers in the method tables to the objects and their VAR sections. The 16-byte header then has to be adjusted for the new linked program.
I gave myself an hour to write a simple linker, and I did it. Then I found that I forgot to recalculate the checksum, but that just took me another 5 minutes to fix. I compiled a "Hello World" program using a stub object in place of the FullDuplexSerial object. I then compiled FullDuplexSerial into its own binary file. I ran the linker on the two binaries, and it produced a single linked binary. It worked the first time.
I'll post the code after I've added a few comments to it. It will take a bit more work to make it handle multiple objects, but I don't think it's a very big task.
Now that is very interesting. If you can write a linker in an hour+5mins, you are the man!
Ok, so this needs a linker and it needs maybe some batch file processing. It should be possible to automate that. And if that is possible, it could potentially be added to BST and Homespun.
I guess your code will reveal all. I'm intrigued by the "stub object".
Nice! I've just spent a day battling to synchronize SD card access with XMM RAM access on the C3, since I currently do it from different cogs. This has advantages in that it doesn't use an extra cog when (for example) a program needs XMM access but doesn't need SD access - but it sure is a pain!
If I end up having to sacrifice a cog anyway to manage the SPI Flash, I may adopt your solution.
Ross.
I have to admit that putting SD card access into the cache driver is a bit tricky. For instance, you have to be able to guarantee that there won't be a cache access while you are reading or writing an SD sector. I've done this by adding code in ZOG to spin on the cache COG's mailbox so there won't be any simultaneous accesses from the cache code.
I added a few comments to the linker program, and have attached the files below. The source file for the program is linkit.c. The zip file also contains a Windows DOS executable, and a batch file, runit.bat, that will link hello1.binary with FullDuplexSerial.binary and produce the file out.binary. I have included the Spin source files hello1.spin and fds_stub.spin.
hello1.spin uses fds_stub.spin instead of FullDuplexSerial.spin. fds_stub.spin contains only the PUB statements from FullDuplexSerial.spin. FullDuplexSerail.binary is the binary file produced by compiling FullDuplexSerial.spin by itself.
linkit.exe will read the parent and child binary files, and produce an output file that combines the two files. The linkit program is limited to a single object in the child binary file, and a single object plus the stub object in the parent binary file. You can modify hello1.spin, compile it to create a new hello1.binary and then run linkit to produce a new output program file.
This version of linkit is just at the proof-of-concept stage. It can be modified to handle more child binaries, or there could be more objects in each binary file. A 64K program could be built by building two separate 32K binaries, and then linking them together. A much larger program could be built from several 32K chunks. Of course, we'll need a new interpreter to run the larger programs.
I added a few comments to the linker program, and have attached the files below.
Interesting.
Maybe I'm missing the use model a little. It seems that stubs refer to real code in a library. I suppose libraries live in external memory or some other media. How does a stub invoke the library? Can you explain the gory details so I can modify my interpreter to handle it?
The stub is used so the compiler generates the correct code. The compiler needs to know the method number in the external object so it generates the correct method number for the call. The compiler also verifies that the number of arguments matches between the calling routine and the target method, so the stub needs to contain the correct number of arguments for each method. The extra stack variables aren't required in the stub, since this adjustment is made at run time.
The point is that we can use the current compilers to produce binaries that are greater than 32K. Each chunk needs to fit in 32K, but the final binary could be mega-bytes in size. The libraries are linked together with the other objects to provide a single binary. The stub doesn't invoke a library, but is replaced by the library at link time.
This concept is a bit different than I proposed earlier in this thread. My initial proposal retained the stub methods, which would perform a jump to the library. That would actually allow for dynamic linking of objects at run time, but that seems more complicated than needed.
Edit: Try adding the following lines to hello1.spin
var
byte bigbuf[32550]
This will be too big with FullDuplexSerial, but it will build OK with fds_stubs. You can then use linkit to link it to FullDuplexSerial.binary. Of course, this will be too big to run in a Prop because the stack will start at $8274, which is at the beginning of the ROM. I'll modify SpinSim to have 64K of RAM. It should run without any other changes.
Ok, so I should just be able to put the linkit binary into external memory and load/execute that normally via cache as long as pcurr, vbase, dbase, etc... are accessed as longs?
Steve, that should work if your Spin interpreter can execute out of external memory.
I just realized that SpinSim supports 64K without any changes. It has 64K of RAM with the ROM image copied into the second 32K. I don't check if the ROM area is being written into, so it can be used just like the first 32K of RAM. The only thing is that the runner code from $FF70 to $FFFF is needed for cognew/coginit of a Spin cog, and the Spin interpreter starting at $F002 is needed if the -p option is used. So there is 60K available with the -p option and almost the entire 64K is available without the -p option.
I linked the large version of hello1 with conio.binary, and the stack starts at $80FC. It ran OK under SpinSim. conio.spin has the same methods as FullDuplexSerial except for an extra "out" method at the end, so I didn't need to create a different stub to use conio.
I managed to get the spinsim016 hello example to compile and run in the simulator. Load took a while, but that's fine.
I tried a more complicated example with the spinsim016 demo, and the linker apparently does not support multiple child binaries just yet. Do you have an updated source for that?
So first question - are you creating that file manually or automatically? If the latter, is it just a matter of going through the file and looking for any line that starts with PUB or PRI?
Then I see you make a big file. That sits entirely in external memory, right?
I imagine there could be a discussion about whether you have a stack in external memory as well, or whether it is in the hub. Leaving that aside for the moment, assume that big binary file is 50k in size and sitting in external memory, eg in the C3's ram chips.
How do you now run this? Is it going to need some sort of custom code to get longs from the external memory and feed them to a spin interpreter?
I managed to get the spinsim016 hello example to compile and run in the simulator. Load took a while, but that's fine.
I tried a more complicated example with the spinsim016 demo, and the linker apparently does not support multiple child binaries just yet. Do you have an updated source for that?
linkit only allows for one child. It will merge the first object from the parent with the first object from the child. It would be fairly easy to change it so that it will merge all of the objects from the parent except the last one with all of the objects of the child. Basically, it would remove the last object from the parent, which is the stub object, and replace it by the child. This would allow merging multiple children in multiple steps. The parent is merged with the first child. The output of this would then be merged with the second child, and so on. This will work only if the stub object is the last object. If the stub object doesn't automatically come out to be the last object a dummy reference to the stub object would need to be added in the last object. I'll try to get that done tonight.
So first question - are you creating that file manually or automatically? If the latter, is it just a matter of going through the file and looking for any line that starts with PUB or PRI?
Then I see you make a big file. That sits entirely in external memory, right?
I imagine there could be a discussion about whether you have a stack in external memory as well, or whether it is in the hub. Leaving that aside for the moment, assume that big binary file is 50k in size and sitting in external memory, eg in the C3's ram chips.
How do you now run this? Is it going to need some sort of custom code to get longs from the external memory and feed them to a spin interpreter?
I created the stub manually. It would be pretty easy to write a utility that could extract the PUB and PRI lines to produce the stub file. It would need to understand Spin comments, so it doesn't extract a PUB/PRI that is commented out. The big file would need to sit in external memory if it is larger than 32K. Spin has three main memory spaces, which are pointed to by PBASE, VBASE and DBASE. PBASE is the beginning of program space. This is most likely in external RAM. VBASE is the beginning of the VAR data, which could be in hub RAM or external RAM depending on the size. DBASE is the beginning of the stack frame. Spin uses the stack on almost every instruction, so it would be best to have the stack in hub RAM. The memory spaces can be manipulated by adjusting the values of PBASE, VBASE and DBASE.
This would need to be run by jazzed's BigSpin interpreter. linkit uses the standard Spin program header, which means that programs are limited to 64K in size. PBASE, VBASE and DBASE are all long addresses -- i.e., the two least significant bits are always zero. We could right-shift these by two to handle a memory space of 256K. The header contains two other values -- PCURR and DCURR, which are the program starting address and the stack starting address. DCURR is a long address, so it can be right shifted by two also. I believe PCURR is a byte address, so it cannot be right shifted. We could make it relative to PBASE. That means that the starting address would have to be within 64K of PBASE. The method tables are OK as is, except for the object entries. We would have to right-shift these entries.
Of course, another approach would be to replace the 16-bit entries with 24-bit or 32-bit entries.
Of course, another approach would be to replace the 16-bit entries with 24-bit or 32-bit entries.
Would it all still glue together properly? I'm thinking that as soon as you make it 64k, that won't be enough. Nor 256k. Is it worth going straight to 32 bit? Is that hard to do?
For me this stub objects look like header files in C. So why not just call theme the same name as the real object but with an additional .h.
The stub for "FullDuplexSerial.spin" would be "FullDuplexSerial.h.spin" and when you include it in the object section you write:
OBJ ser : "FullDuplexSerial.h"
Then the linker has the information about the real object file to include, by just let the .h away.
And for generating such "Header" files:
Load the real object in PropTool and click on the "Summary" option, then copy the resulting text to a new file. I think we have also to add the constants from the CON sections to the stub files. This must be made manually in this case.
I haven't updated linkit yet because I wanted to think a bit more about the best approach. I've decided to keep it simple and still replace only one child stub at a time. However, I'll add a parameter so that the object number for the stub can be specified. The default number will be the last stub.
As an example, the OBJ section of demo.spin in the SpinSim directory looks like this:
obj
ser : "conio"
fs : "fileio"
This could be replaced by stub objects as follows:
obj
ser : "conio_stub"
fs : "fileio_stub"
After I update linkit the program would be linked with the following commands:
The "-2" wouldn't have to be specified since it would be the default in this case. The top object will be treated as the zero-th object number, and a -0 would not be allowed. Keep in mind that the object numbering can get complicated. If conio.binary contained more than one object fileio_stub would no longer be object number 2 after conio.binary was linked in. Also, if fileio included conio as an object the Prop tool would make it object number 1 and conio would become object number 2. The Prop tool shows a diagram of the object ordering when a compile is performed.
Another thing I should point out is that multiple copies of an object may exist after a link is performed. In the preceding exampe, if fileio referenced conio, then its binary would contain a copy of conio in it. This would create two instances of VAR data, which could cause a problem if the object assumed there was only one instance. It would be a problem for an object like FullDuplexSerial that keeps its state information in VAR data.
Dr_Acula, after giving it some thought I agree that it's probably better to just use 32-bit values, so I'll add that as an option. We can distinguish between the standard 16-bit header and the 32-bit header because the value of PBASE will be 26 instead of 16. The BigSpin interpreter will need to handle the different header format.
Ariba, the stub will look like a C header file that contains #define's and function prototypes. However, I prefer tacking on a "_stub" to the file name instead of ".c". The utility that extracts the PUB/PRI lines could also extract the CON entries.
However, I prefer tacking on a "_stub" to the file name instead of ".c". The utility that extracts the PUB/PRI lines could also extract the CON entries.
How about "conio_lib.spin" ? Doesn't really matter and stub is more specific, but the file is after all referring to a library.
Let me know what the 32 bit format will look like. It will complicate things. I guess just replacing the words with longs would be fine. I'm still debugging the interpreter for some issues (among many other things ) so i'm not in a hurry for an updated format.
I created the stub manually. It would be pretty easy to write a utility that could extract the PUB and PRI lines to produce the stub file. It would need to understand Spin comments, so it doesn't extract a PUB/PRI that is commented out.
I threw together some quick code to do this. FullDuplexSerial.spin is in a slightly unusual format - rename it as .bin and look at it and every alternate character is a zero, and of course there might be characters from 128 to 255 eg as part of a schematic. So need to strip all these out and just work with the text bytes and only save lines that start with PUB or PRI.
Dim LineofText As String
Dim Textarray(30000) As String
Dim FileLength As Integer
Dim FileCounter As Integer
Dim i As Long
Dim FileNamePath As String
Dim BinaryFileLength As Long
Dim BinaryFileCounter As Long
Dim b As Byte
' spin files are in an unusual format, some characters >127 and many zeros
FileNamePath = TextBox20.Text ' c:\testfile.spin
Dim FileRead As New FileStream(FileNamePath, FileMode.Open, FileAccess.Read)
Dim br As New BinaryReader(FileRead) ' binary reader
LineofText = ""
BinaryFileLength = br.BaseStream.Length() - 1 ' get binary file length
For i = 0 To BinaryFileLength
b = br.ReadByte() ' read the byte
If b >= 32 And b <= 127 Then ' discard all special characters
LineofText += Strings.Chr(b) ' add to line
End If
If b = 13 Then ' carriage return
Textarray(FileCounter) = LineofText ' store the line
FileCounter += 1 ' add one to counter
LineofText = "" ' clear the line
End If
Next
FileRead.Close() ' close the input file
FileLength = FileCounter - 1 ' get the file length
FileOpen(1, TextBox21.Text, OpenMode.Output) ' open the output file
For i = 0 To FileLength
LineofText = Textarray(i) ' get the line
If Strings.Left(LineofText, 3) = "PUB" Or Strings.Left(LineofText, 3) = "PRI" Then
PrintLine(1, LineofText) ' save if starts with PUB or PRI
End If
Next i
FileClose(1)
Steve, I think the 32-bit header would look the same as the 16-bit header for the first 6 bytes, which are the clock rate, clock mode and checksum. The remaining 10 bytes, or 5 words, would become 5 longs instead. This would make the header 26 bytes long instead of 16 bytes, so the top object's starting address will be 26 ($1A) instead of 16 ($10). The object entries in the method tables would need to increase also. It will be a while before I get it sorted out.
I did some more work on linkit, and I almost have it where it can replace a stub object in the middle of the parent object instead at the end. I'm still trying to get the VAR adjustment correct. I hope to have a 16-bit version done by tomorrow, and then add 32-bit support later.
Dr_Acula, that program looks good from extracting the PUB and PRI statements. Spin files that contain special characters use a 16-bit character set, which is the reason for the zero bytes. You should do a case insensitive compare to handle the use of pub, pri, Pub, Pri and so on. You'll also have to handle cases where PUB/PRI statement are within a commented area so that they get ignored. Other objects can reference the constants set in the CON section, so those would be needed in the general case. You should not include the VAR data in the stub. It makes it easier if I no the stub has no VAR data.
Comments
Hi Jazzed,
Do you have any actual statistics on this yet?
The reason I ask is that it looks to me very much like I will not be able to fit the additional code required for handling Flash RAM into the current Catalina XMM Kernel - so it may be necessary to move the XMM access code out to another cog. If I have to do this, I would probably adopt your caching driver API, since I believe caching the serial RAM/Flash access may also result in a substantial performance boost - but this is mainly because serial access is many, many times slower than parallel access. On a board with reasonably fast parallel access to the SRAM, some (perhaps all!) of the benefit of caching will be lost due to the the additional synchronization required to coordinate activities between the kernel and the caching driver.
If you have any actual data on this, I'd be really interested to see it.
Ross.
I think that is the key - and if you have code already working then problem solved. It is now abstracted to a simple command to read and one to write.
The C3 probably is the hardware that BigSpin is going to run on for most people, so this is great. And for any different hardware, like you say, replace the object.
So ok, you can read and write longs to a flat memory space.
It seems to me you would not want to restrict that to the memory size of the C3. Nor even to 512k or even bigger memory chips. Given the prop is a 32 bit micro, it seems logical to choose a 32 bit memory space. Do that for the simulator, and use that for the code as well, and if you tell the compiler what actual memory you have, it can tell you the % you have used as you add to the code.
Dave seems to be moving ahead very rapidly with this. I'm intrigued with the the possibility that existing compilers could do most of the work, and the only 'new' code might be a linker.
You don't have to synchronize for every access, if that were required I would agree with you on lost benefit.
I'm too busy at this point to create code that would take approximately 4us to deliver one byte of data when 32 times that data can be delivered in only 3 times the wait. I might do it later as an academic exercise.
I don't have a Drac board to do such an experiment, but I know others do ;-)
The cache sounds like the answer to the memory problem. Code is already written and it works. Kudos to jazzed.
Going back to something Dave said
This seems very logical. Does the same apply with some code you see on the obex where there are child processes with their own child processes? Not all code is two or more layers deep but some is.
More likely though, code in an obex is a group of pasm and spin, and if you can abstract that to a block, with pointers to go in and out of that block, it ought to decrease the modifications needed to obex code. Fantastic if indeed this is easy to implement, because this could be the last piece of the jigsaw.
Ross, I forgot to mention that David Betz has already done this for C3 for you to use and his code should be easily integrated. If you add this, Catalina will automatically support SDRAM solutions as well. Zog may get jealous.
Hi Jazzed,
Yes, I know about David's work for the C3 - so I guess I only have to wait to find out the answer to my question. But I was thinking more generally about whether it would be a good idea to adopt a caching solution across all platforms - but my concern is that on some platforms this may actually be detrimental.
When I get some time, I'll do some experiments of my own.
Ross.
Ummm... I think you're giving me a little more credit than is due! :-)
Actually, the C3 cache code as it currently stands is really two separate caches. One is used to cache accesses to the 1mb SPI flash chip and the other is used to cache access to the 64k of SPI SRAM. Both caches are direct mapped caches. Since the GCC linker script places code in flash and data in SRAM this could be considered separate instruction and data caches. I have considered doing a unified set associative cache but haven't had time to work out an implementation.
Also, the main reason that the code in the C3 cache COG is bigger than the code in the SDRAM cache COG is that it includes code to erase and write the flash and also code for SD card sector I/O. This is to allow all of the C3 SPI devices (minus the A/D chip) to be managed by a single COG.
Can you explain that a bit more? Ok, my simplistic thinking it that you have something in the obex. It has a whole lot of spin PUBs and it might have some pasm code. If you hit 'compile' that obex code ought to compile to something, even if it can't run. The dat and the var data I think ends up concatenated and at the beginning. Then there is the PUB code.
So my understanding is you do something clever to that code. Do you create a list of calls to PUBs and group all those together so there is a list where everything is in order? Or in more general terms, how do you know where the start of each PUB has ended up?
I imagine if you do create some self contained code, with known locations for each PUBs and the code is internally consistent then you can use that in some way by a higher level 'main' program.
Is this the idea or am I getting a bit muddled?!
Nice! I've just spent a day battling to synchronize SD card access with XMM RAM access on the C3, since I currently do it from different cogs. This has advantages in that it doesn't use an extra cog when (for example) a program needs XMM access but doesn't need SD access - but it sure is a pain!
If I end up having to sacrifice a cog anyway to manage the SPI Flash, I may adopt your solution.
Ross.
A Spin object consists of a method table, DAT section and PUB/PRI code. A binary file contains a 16-byte header that is used to set up the clock and the various pointers used by the spin interpreter. The VAR sections for all of the objects follow immediately after the last PUB/PRI method in the last object, and the stack begins after that. PASM code is treated as regular DAT data, and Spin doesn't really know the difference between PASM code and any other DAT data.
The linker just has to concatenate all of the objects together and fix up the pointers in the method tables to the objects and their VAR sections. The 16-byte header then has to be adjusted for the new linked program.
I gave myself an hour to write a simple linker, and I did it. Then I found that I forgot to recalculate the checksum, but that just took me another 5 minutes to fix. I compiled a "Hello World" program using a stub object in place of the FullDuplexSerial object. I then compiled FullDuplexSerial into its own binary file. I ran the linker on the two binaries, and it produced a single linked binary. It worked the first time.
I'll post the code after I've added a few comments to it. It will take a bit more work to make it handle multiple objects, but I don't think it's a very big task.
Dave
Ok, so this needs a linker and it needs maybe some batch file processing. It should be possible to automate that. And if that is possible, it could potentially be added to BST and Homespun.
I guess your code will reveal all. I'm intrigued by the "stub object".
hello1.spin uses fds_stub.spin instead of FullDuplexSerial.spin. fds_stub.spin contains only the PUB statements from FullDuplexSerial.spin. FullDuplexSerail.binary is the binary file produced by compiling FullDuplexSerial.spin by itself.
linkit.exe will read the parent and child binary files, and produce an output file that combines the two files. The linkit program is limited to a single object in the child binary file, and a single object plus the stub object in the parent binary file. You can modify hello1.spin, compile it to create a new hello1.binary and then run linkit to produce a new output program file.
This version of linkit is just at the proof-of-concept stage. It can be modified to handle more child binaries, or there could be more objects in each binary file. A 64K program could be built by building two separate 32K binaries, and then linking them together. A much larger program could be built from several 32K chunks. Of course, we'll need a new interpreter to run the larger programs.
Maybe I'm missing the use model a little. It seems that stubs refer to real code in a library. I suppose libraries live in external memory or some other media. How does a stub invoke the library? Can you explain the gory details so I can modify my interpreter to handle it?
The point is that we can use the current compilers to produce binaries that are greater than 32K. Each chunk needs to fit in 32K, but the final binary could be mega-bytes in size. The libraries are linked together with the other objects to provide a single binary. The stub doesn't invoke a library, but is replaced by the library at link time.
This concept is a bit different than I proposed earlier in this thread. My initial proposal retained the stub methods, which would perform a jump to the library. That would actually allow for dynamic linking of objects at run time, but that seems more complicated than needed.
Edit: Try adding the following lines to hello1.spin This will be too big with FullDuplexSerial, but it will build OK with fds_stubs. You can then use linkit to link it to FullDuplexSerial.binary. Of course, this will be too big to run in a Prop because the stack will start at $8274, which is at the beginning of the ROM. I'll modify SpinSim to have 64K of RAM. It should run without any other changes.
I just realized that SpinSim supports 64K without any changes. It has 64K of RAM with the ROM image copied into the second 32K. I don't check if the ROM area is being written into, so it can be used just like the first 32K of RAM. The only thing is that the runner code from $FF70 to $FFFF is needed for cognew/coginit of a Spin cog, and the Spin interpreter starting at $F002 is needed if the -p option is used. So there is 60K available with the -p option and almost the entire 64K is available without the -p option.
I linked the large version of hello1 with conio.binary, and the stack starts at $80FC. It ran OK under SpinSim. conio.spin has the same methods as FullDuplexSerial except for an extra "out" method at the end, so I didn't need to create a different stub to use conio.
I managed to get the spinsim016 hello example to compile and run in the simulator. Load took a while, but that's fine.
I tried a more complicated example with the spinsim016 demo, and the linker apparently does not support multiple child binaries just yet. Do you have an updated source for that?
Thanks.
I think I can see what you are doing. This is your stub file
PUB start(rxpin, txpin, mode, baudrate) : okay PUB stop PUB rxflush PUB rxcheck : rxbyte PUB rxtime(ms) : rxbyte | t PUB rx : rxbyte PUB tx(txbyte) PUB str(stringptr) PUB dec(value) | i, x PUB hex(value, digits) PUB bin(value, digits)Which came from fullduplexserial.
So first question - are you creating that file manually or automatically? If the latter, is it just a matter of going through the file and looking for any line that starts with PUB or PRI?
Then I see you make a big file. That sits entirely in external memory, right?
I imagine there could be a discussion about whether you have a stack in external memory as well, or whether it is in the hub. Leaving that aside for the moment, assume that big binary file is 50k in size and sitting in external memory, eg in the C3's ram chips.
How do you now run this? Is it going to need some sort of custom code to get longs from the external memory and feed them to a spin interpreter?
I created the stub manually. It would be pretty easy to write a utility that could extract the PUB and PRI lines to produce the stub file. It would need to understand Spin comments, so it doesn't extract a PUB/PRI that is commented out. The big file would need to sit in external memory if it is larger than 32K. Spin has three main memory spaces, which are pointed to by PBASE, VBASE and DBASE. PBASE is the beginning of program space. This is most likely in external RAM. VBASE is the beginning of the VAR data, which could be in hub RAM or external RAM depending on the size. DBASE is the beginning of the stack frame. Spin uses the stack on almost every instruction, so it would be best to have the stack in hub RAM. The memory spaces can be manipulated by adjusting the values of PBASE, VBASE and DBASE.
This would need to be run by jazzed's BigSpin interpreter. linkit uses the standard Spin program header, which means that programs are limited to 64K in size. PBASE, VBASE and DBASE are all long addresses -- i.e., the two least significant bits are always zero. We could right-shift these by two to handle a memory space of 256K. The header contains two other values -- PCURR and DCURR, which are the program starting address and the stack starting address. DCURR is a long address, so it can be right shifted by two also. I believe PCURR is a byte address, so it cannot be right shifted. We could make it relative to PBASE. That means that the starting address would have to be within 64K of PBASE. The method tables are OK as is, except for the object entries. We would have to right-shift these entries.
Of course, another approach would be to replace the 16-bit entries with 24-bit or 32-bit entries.
Dave
Would it all still glue together properly? I'm thinking that as soon as you make it 64k, that won't be enough. Nor 256k. Is it worth going straight to 32 bit? Is that hard to do?
The stub for "FullDuplexSerial.spin" would be "FullDuplexSerial.h.spin" and when you include it in the object section you write:
OBJ ser : "FullDuplexSerial.h"
Then the linker has the information about the real object file to include, by just let the .h away.
And for generating such "Header" files:
Load the real object in PropTool and click on the "Summary" option, then copy the resulting text to a new file. I think we have also to add the constants from the CON sections to the stub files. This must be made manually in this case.
Andy
As an example, the OBJ section of demo.spin in the SpinSim directory looks like this: This could be replaced by stub objects as follows: After I update linkit the program would be linked with the following commands:
linkit hello.binary -1 conio.binary out.binary
linkit out.binary -2 fileio.binary out.binary
The "-2" wouldn't have to be specified since it would be the default in this case. The top object will be treated as the zero-th object number, and a -0 would not be allowed. Keep in mind that the object numbering can get complicated. If conio.binary contained more than one object fileio_stub would no longer be object number 2 after conio.binary was linked in. Also, if fileio included conio as an object the Prop tool would make it object number 1 and conio would become object number 2. The Prop tool shows a diagram of the object ordering when a compile is performed.
Another thing I should point out is that multiple copies of an object may exist after a link is performed. In the preceding exampe, if fileio referenced conio, then its binary would contain a copy of conio in it. This would create two instances of VAR data, which could cause a problem if the object assumed there was only one instance. It would be a problem for an object like FullDuplexSerial that keeps its state information in VAR data.
Dr_Acula, after giving it some thought I agree that it's probably better to just use 32-bit values, so I'll add that as an option. We can distinguish between the standard 16-bit header and the 32-bit header because the value of PBASE will be 26 instead of 16. The BigSpin interpreter will need to handle the different header format.
Ariba, the stub will look like a C header file that contains #define's and function prototypes. However, I prefer tacking on a "_stub" to the file name instead of ".c". The utility that extracts the PUB/PRI lines could also extract the CON entries.
How about "conio_lib.spin" ? Doesn't really matter and stub is more specific, but the file is after all referring to a library.
Let me know what the 32 bit format will look like. It will complicate things. I guess just replacing the words with longs would be fine. I'm still debugging the interpreter for some issues (among many other things
I threw together some quick code to do this. FullDuplexSerial.spin is in a slightly unusual format - rename it as .bin and look at it and every alternate character is a zero, and of course there might be characters from 128 to 255 eg as part of a schematic. So need to strip all these out and just work with the text bytes and only save lines that start with PUB or PRI.
Dim LineofText As String Dim Textarray(30000) As String Dim FileLength As Integer Dim FileCounter As Integer Dim i As Long Dim FileNamePath As String Dim BinaryFileLength As Long Dim BinaryFileCounter As Long Dim b As Byte ' spin files are in an unusual format, some characters >127 and many zeros FileNamePath = TextBox20.Text ' c:\testfile.spin Dim FileRead As New FileStream(FileNamePath, FileMode.Open, FileAccess.Read) Dim br As New BinaryReader(FileRead) ' binary reader LineofText = "" BinaryFileLength = br.BaseStream.Length() - 1 ' get binary file length For i = 0 To BinaryFileLength b = br.ReadByte() ' read the byte If b >= 32 And b <= 127 Then ' discard all special characters LineofText += Strings.Chr(b) ' add to line End If If b = 13 Then ' carriage return Textarray(FileCounter) = LineofText ' store the line FileCounter += 1 ' add one to counter LineofText = "" ' clear the line End If Next FileRead.Close() ' close the input file FileLength = FileCounter - 1 ' get the file length FileOpen(1, TextBox21.Text, OpenMode.Output) ' open the output file For i = 0 To FileLength LineofText = Textarray(i) ' get the line If Strings.Left(LineofText, 3) = "PUB" Or Strings.Left(LineofText, 3) = "PRI" Then PrintLine(1, LineofText) ' save if starts with PUB or PRI End If Next i FileClose(1)output is this
Did you say you wanted to include some VAR data as well?
This sort of text processing ought to be possible in any one of a number of languages.
I did some more work on linkit, and I almost have it where it can replace a stub object in the middle of the parent object instead at the end. I'm still trying to get the VAR adjustment correct. I hope to have a 16-bit version done by tomorrow, and then add 32-bit support later.
Dr_Acula, that program looks good from extracting the PUB and PRI statements. Spin files that contain special characters use a 16-bit character set, which is the reason for the zero bytes. You should do a case insensitive compare to handle the use of pub, pri, Pub, Pri and so on. You'll also have to handle cases where PUB/PRI statement are within a commented area so that they get ignored. Other objects can reference the constants set in the CON section, so those would be needed in the general case. You should not include the VAR data in the stub. It makes it easier if I no the stub has no VAR data.