Shazz said...
@CosmicBob : funny extract [noparse]:)[/noparse] Sometimes I think I recognize words from the hidden part scroll text [noparse]:)[/noparse] but I guess that's an optical illusion [noparse]:)[/noparse] But definitively it looks like packed data as the size is smaller than real text (comparing to obfuscation/encryption)
I forgot to mention that I extracted 5 characters or more and nothing else. I'll re-do and take everything just it in case the first extraction is misleading in some regard.
Bye the way, I also put it into a language converter to see if some of the text was in Swedish but it' look the same (at a quick glance)
Both of my prop arrangements have 64K eeroms, only because of Ybox2 experiments. On my boards the Tubulence logo glitches very slightly but the demo video doesn't.
@KPR : I guess yes it should work easily with any EEPROM size as it is used as a mass storage and no direct relation to the RAM size... But if we don't mind of the compatibility (I mean most of prop owners seem to have 32 Kb EEPROM), better to use the SD adapter no ? (I don't know how much different it can be in terms of speed, I mean reading from the EEPROM vs. the SD Card, both use the prop I/Os so it should be inherent to the EEPROM and the SD speed ?)
Chris Micro said...
A very good way to understand the compression tecnics and other things used is to take a look on the source code of Craft Project from Linus:
Eh eh yep I did that too and look at the readchunck(ee) label in music.S... I did not yet really understand how this depacker works [noparse][[/noparse]"the unpacker represented by an eeprom pointer in Z "] but I did not find the real depacker code yet... Did you understand more than I how it works ?
Shazz said...
@KPR : I guess yes it should work easily with any EEPROM size as it is used as a mass storage and no direct relation to the RAM size... But if we don't mind of the compatibility (I mean most of prop owners seem to have 32 Kb EEPROM), better to use the SD adapter no ? (I don't know how much different it can be in terms of speed, I mean reading from the EEPROM vs. the SD Card, both use the prop I/Os so it should be inherent to the EEPROM and the SD speed ?)
I was just figuring that a larger eeprom would be a logical choice since the code is already there and being used, as well as the 2 pins to reference it.. an sd reader would take up cogs / ram / pins.. So you don't get as much space, but resource wise it seems to be a better choice.. What do you think..
k.
Man now I'm really excited in seeing the code from this demo, It reminds me of when I spent weeks on dissembling the Crackto on the Apple ][noparse][[/noparse] release of Ultima 3 by Crustaceo Mutoid..
@KPR : yep right... in terms of resources that's a really logical choice after all.. But the way I just ordered the SD adapter from Brian, I need to check how much resource it needs.
And yes.... I felt the same while depacking then disassembling Atari ST cracktros... [noparse]:)[/noparse] But for the moment, in terms of tools (debugger, disassembler,...) it looks more difficult on the Propeller... I'd like to find a way to dump the cogs RAM on demand and stuff like that...
But I'm sure we'll have to prove we deserve the sourcecode ! And a fist step is to understand clearly the depacker to be able to depack the whole demo first (would be sooo easier to dump the RAMs while executing but I don't know how to do that !).
Shazz said...
@KPR : yep right... in terms of resources that's a really logical choice after all.. But the way I just ordered the SD adapter from Brian, I need to check how much resource it needs.
And yes.... I felt the same while depacking then disassembling Atari ST cracktros... [noparse]:)[/noparse] But for the moment, in terms of tools (debugger, disassembler,...) it looks more difficult on the Propeller... I'd like to find a way to dump the cogs RAM on demand and stuff like that...
But I'm sure we'll have to prove we deserve the sourcecode ! And a fist step is to understand clearly the depacker to be able to depack the whole demo first (would be sooo easier to dump the RAMs while executing but I don't know how to do that !).
Well looks like I have some reading to do.. man its been 20 plus years since I played with assembly, god I feel old all of a sudden, 6502 assembly anyone..
There has to be a way to dump cog ram, you should post on its own and see what that brings up.. I have been watching this thread like a hawk since I seen the demo, off hand where you part of TSi?
The COGs work completely independent of one another. There is no dumping of COG ram on demand, unless a bit of wedge code is inserted into the targeted program.
Once launched, their program area is filled from the designated HUB ram, and they start running. There is then no association between HUB ram, and any other COG ram.
This code is going to have to be parsed, starting with the EEPROM image load. We know what the propeller does on boot, and we have the image. That's it.
I think this is interesting, given all the threads we've had on protected code!! The result of those was making it difficult for others to see the code and that there is no perfect security. It appears that a very difficult puzzle can be made of the EEPROM image! Doing something like this and perhaps potting the chip is going to be decent protection.
@potatohead - looks like I have to go back to the forums for some more reading now.. It would be nice if Linus were to give some hints.. you know dangle the carrot in front of the donkey so to speak..
potatohead said...
The COGs work completely independent of one another. There is no dumping of COG ram on demand, unless a bit of wedge code is inserted into the targeted program.
Once launched, their program area is filled from the designated HUB ram, and they start running. There is then no association between HUB ram, and any other COG ram.
This code is going to have to be parsed, starting with the EEPROM image load. We know what the propeller does on boot, and we have the image. That's it.
I think this is interesting, given all the threads we've had on protected code!! The result of those was making it difficult for others to see the code and that there is no perfect security. It appears that a very difficult puzzle can be made of the EEPROM image! Doing something like this and perhaps potting the chip is going to be decent protection.
LOL!!! Wonder if Linus read those threads too?
Thanks for all those informations Mr Patate [noparse]:)[/noparse] I try to read as many threads as possible from this forum but ouch, that's a lot ! So ok.... I thought it could be a good idea to start writing a kind of loader (quite similar to Chip's booter) to be set in RAM then in a Cog that can really does like the booter (Load EEPROM,...) but stay in a cog and watch, dump what's happening.
So I understand that's not so easy and I'll stick on disassembling the EEPROM and I hope to find+understand the depacker. For the moment I never thought that Linus tried to protect his code, only a packer in my point of view... Bah true or not, the challenge is still fun !
Dunno if Linus read that but as he started the challenge.... I guess yes.. a little... to have fun [noparse]:)[/noparse]
I haven't done any disassembly, but this is how I would do it:
The initial SPIN code just does a COGINIT( COG, @bootloader, @PARin) which puts the Prop in all-PASM mode. Linus can now use all 32K of HUB RAM. The bootloader is a decompressor, and it decompresses a chunk of HUB RAM, overwritting a "sacrificial" part of the initial 32K load. The bootloader then COGINIT's the other 7 COGs (and maybe itself) from the decompressed data. The HUB RAM is then reloaded from the EEPROM, and the second chunk is decompressed. Then everything kicks off.
Like potatohead said, in order to understand it you're going to have to walk the boot process just like crackers did in the Apple][noparse][[/noparse] & C64 days.
Eric.... you're more experienced than I am (so please tell me why I'm wrong, I need it !) but I would rather have imagined a flow which bypasses as much as possible the Hub RAM (and keep it for the VGA buffers and bitmaps and ok a little temporary (2K) buffer to unpack code to be transfer to a cog).
I mean, as you said the bootloader is a depacker, so a Cog is reserved to depack on demand.
But then, in my mind the depacker will first read some bytes from the EEPROM (and not from RAM which was cleared by the loader), depack them (one effect for example) in the hub RAM buffer, loop until the end of the block (code+data for the given effect), then coginit it to an available COG. And continue for the next effect and so on.
So in my case the data are really coming from the EEPROM and the Hub will only acts as a temporary buffer.
I thought it could be done like that as the demo doesn't work if the EEPROM is not containing the demo.
makes senses ???
Oh, about disassembly, I'm about to write my own disassembler as I cannot find one which works.... Did I miss something ? (before wasting some time)
In my opinion COGs loads with first LOAD.
Then RUN decoding from EEProm and display Display it from HUB
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔ Nothing is impossible, there are only different degrees of difficulty. For every stupid question there is at least one intelligent answer. Don't guess - ask instead. If you don't ask you won't know. If your gonna construct something, make it·as simple as·possible yet as versatile as posible.
Working with the fact that the whole of the 32KB EEProm gets copied to Hub Ram.
What I think is that different parts of the demo which are 'packed' are overwritten in the Hub Ram when they are 'unpacked' therefore a copy of them needs to be retained in theEEProm for later use.
When they are needed they are loaded into Hub Ram from the EEProm and 'unpacked' again overwriting the Hub Ram.
A similar technique we use with PropGFX
Regards,
Coley
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔ PropGFX - The home of the Hybrid Development System and PropGFX Lite
If no host communication was detected, the Boot
Loader looks for an external 32 KB EEPROM on
pins P28 and P29. If an EEPROM is detected,
the entire 32 KB data image is loaded into the
Propeller chip’s global RAM.
So the EEProm image may also contain stack space for the VGA framebuffer that makes it up to 32K.
I doubt that Linus would be that wasteful, instead I think different parts of the demo are unpacked from the EEProm and they are loaded into cogs and run from there leaving as much possible free ram for the display.
Remember Linus clearly states at the beginning of the demo that the Propeller has 48K Ram, this implies (to me) he is using the COG Ram to full effect.
BEEP said...
I think there are 3 video layers/3 cogs sending on PIN 0..2 to a cog which sends out the VGA signal.
On my demoboard P0 and P1 are toast, so I only get one of the video layers displayed.
Do an experiment pull some of your pins P0-P7 down to ground and see what effect it has, you will be enlightened.
Regards,
Coley
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔ PropGFX - The home of the Hybrid Development System and PropGFX Lite
If you have the hardware, my Propalyzer application lets you see all the activity except pins 30&31. You'll have to set triggers for state changes though. Looks like Linus is constantly reading EEPROM ... of course you can see that with a scope.
and watching pin 4 to 7 on my 1 channel oscilloscope was a bit boring and time-consuming (20 minutes).
LOL ! I just watched an episode of Big Bang Theory and I could have heard that ! [noparse]:D[/noparse]
Jazzed said...
If you have the hardware, my Propalyzer application lets you see all the activity except pins 30&31. You'll have to set triggers for state changes though. Looks like Linus is constantly reading EEPROM ... of course you can see that with a scope.
ah !!! So my idea was the good one ! And so it explains/confirms the need of the demo in EEPROM (during all the time of the demo) to run. Thanks Jazzed, I feel better [noparse]:)[/noparse]
btw, I nearly finished to code my PASM disassembler. I hope to post it there tomorrow. Maybe it would help.
Shazz said...
Eric.... you're more experienced than I am (so please tell me why I'm wrong, I need it !) but I would rather have imagined a flow which bypasses as much as possible the Hub RAM (and keep it for the VGA buffers and bitmaps and ok a little temporary (2K) buffer to unpack code to be transfer to a cog).
I mean, as you said the bootloader is a depacker, so a Cog is reserved to depack on demand. But then, in my mind the depacker will first read some bytes from the EEPROM (and not from RAM which was cleared by the loader), depack them (one effect for example) in the hub RAM buffer, loop until the end of the block (code+data for the given effect), then coginit it to an available COG. And continue for the next effect and so on.· So in my case the data are really coming from the EEPROM and the Hub will only acts as a temporary buffer. I thought it could be done like that as the demo doesn't work if the EEPROM is not containing the demo. makes senses ???
Heh, you're right.· There's no reason to load the EEPROM to HUB RAM before decompressing it since the data from the EEPROM has to flow through a COG to get to HUB RAM in the first place.· Unless the EEPROM timing/space requirements means on the fly decompression isn't feasible.· Also, byte access is easier from HUB RAM, which may be valuable depending on the decompression algorithm.
1. Using the IDE, save the EEPROM as .binary
2. run the demo in EEPROM (power one so)
3. while the demo is running, using ViewPort, run the binary (so in RAM)
4. the hidden screen appear.
Don't ask me why, I found it by mistake [noparse]:)[/noparse] I don't know why ViewPort loads the stuff differently than the Prop IDE...
tonyp12 said...
Fact: All of his code is compressed except the first (guestimated) 80bytes
All of the 32kb eeprom gets read in to Hub Ram at boot
though reading the first 80 bytes would be enough, you can not stop the ROMs boot loader.
Only the first 80bytes in RAM is useful, the rest is useless.
The 80 bytes is a decompression routine,
that can read the eeprom from any mem location and any byte lengths.
A demo is a bunch of small programs run in sequence, so you never need all of the code in Hub RAM at one time.
Does he use IBM Code Pack with it's 60% ratio ?? http://www.eecs.umich.edu/~tnm/compress/publications/micro32-slides.pdf
Excellent excellent, we slowly come all together to the same conclusions [noparse]:)[/noparse]
And yep as you said the full EEPROM is loaded by Chip's loader in RAM but I'm quite sure Linus at the beginning coded a clean loop to set the VGA buffers, scratchpad and so on.
And yes, I agree totally the demo is probably a bunch of small COGs programs packed and depacked on demand.
But, why did you say 80 bytes ? did you extract the depacker code ?
I'll take a look at the IBM stuff... but I would bet that linus has written his own packer as he did for Craft.
Comments
http://www.linusakesson.net/scene/craft/index.php
Eh eh yep I did that too and look at the readchunck(ee) label in music.S... I did not yet really understand how this depacker works [noparse][[/noparse]"the unpacker represented by an eeprom pointer in Z "] but I did not find the real depacker code yet... Did you understand more than I how it works ?
mktitle.c is the packer, right ?
Post Edited (Shazz) : 4/17/2009 7:40:06 AM GMT
Post Edited (BEEP) : 4/17/2009 11:34:01 AM GMT
I was just figuring that a larger eeprom would be a logical choice since the code is already there and being used, as well as the 2 pins to reference it.. an sd reader would take up cogs / ram / pins.. So you don't get as much space, but resource wise it seems to be a better choice.. What do you think..
k.
Man now I'm really excited in seeing the code from this demo, It reminds me of when I spent weeks on dissembling the Crackto on the Apple ][noparse][[/noparse] release of Ultima 3 by Crustaceo Mutoid..
And yes.... I felt the same while depacking then disassembling Atari ST cracktros... [noparse]:)[/noparse] But for the moment, in terms of tools (debugger, disassembler,...) it looks more difficult on the Propeller... I'd like to find a way to dump the cogs RAM on demand and stuff like that...
But I'm sure we'll have to prove we deserve the sourcecode ! And a fist step is to understand clearly the depacker to be able to depack the whole demo first (would be sooo easier to dump the RAMs while executing but I don't know how to do that !).
Well looks like I have some reading to do.. man its been 20 plus years since I played with assembly, god I feel old all of a sudden, 6502 assembly anyone..
There has to be a way to dump cog ram, you should post on its own and see what that brings up.. I have been watching this thread like a hawk since I seen the demo, off hand where you part of TSi?
k.
Once launched, their program area is filled from the designated HUB ram, and they start running. There is then no association between HUB ram, and any other COG ram.
This code is going to have to be parsed, starting with the EEPROM image load. We know what the propeller does on boot, and we have the image. That's it.
I think this is interesting, given all the threads we've had on protected code!! The result of those was making it difficult for others to see the code and that there is no perfect security. It appears that a very difficult puzzle can be made of the EEPROM image! Doing something like this and perhaps potting the chip is going to be decent protection.
LOL!!! Wonder if Linus read those threads too?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Propeller Wiki: Share the coolness!
Chat in real time with other Propellerheads on IRC #propeller @ freenode.net
Safety Tip: Life is as good as YOU think it is!
k.
Thanks for all those informations Mr Patate [noparse]:)[/noparse] I try to read as many threads as possible from this forum but ouch, that's a lot ! So ok.... I thought it could be a good idea to start writing a kind of loader (quite similar to Chip's booter) to be set in RAM then in a Cog that can really does like the booter (Load EEPROM,...) but stay in a cog and watch, dump what's happening.
So I understand that's not so easy and I'll stick on disassembling the EEPROM and I hope to find+understand the depacker. For the moment I never thought that Linus tried to protect his code, only a packer in my point of view... Bah true or not, the challenge is still fun !
Dunno if Linus read that but as he started the challenge.... I guess yes.. a little... to have fun [noparse]:)[/noparse]
The initial SPIN code just does a COGINIT( COG, @bootloader, @PARin) which puts the Prop in all-PASM mode. Linus can now use all 32K of HUB RAM. The bootloader is a decompressor, and it decompresses a chunk of HUB RAM, overwritting a "sacrificial" part of the initial 32K load. The bootloader then COGINIT's the other 7 COGs (and maybe itself) from the decompressed data. The HUB RAM is then reloaded from the EEPROM, and the second chunk is decompressed. Then everything kicks off.
Like potatohead said, in order to understand it you're going to have to walk the boot process just like crackers did in the Apple][noparse][[/noparse] & C64 days.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Composite NTSC sprite driver: http://forums.parallax.com/showthread.php?p=800114
I mean, as you said the bootloader is a depacker, so a Cog is reserved to depack on demand.
But then, in my mind the depacker will first read some bytes from the EEPROM (and not from RAM which was cleared by the loader), depack them (one effect for example) in the hub RAM buffer, loop until the end of the block (code+data for the given effect), then coginit it to an available COG. And continue for the next effect and so on.
So in my case the data are really coming from the EEPROM and the Hub will only acts as a temporary buffer.
I thought it could be done like that as the demo doesn't work if the EEPROM is not containing the demo.
makes senses ???
Oh, about disassembly, I'm about to write my own disassembler as I cannot find one which works.... Did I miss something ? (before wasting some time)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Propeller Wiki: Share the coolness!
Chat in real time with other Propellerheads on IRC #propeller @ freenode.net
Safety Tip: Life is as good as YOU think it is!
In my opinion COGs loads with first LOAD.
Then RUN decoding from EEProm and display Display it from HUB
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nothing is impossible, there are only different degrees of difficulty.
For every stupid question there is at least one intelligent answer.
Don't guess - ask instead.
If you don't ask you won't know.
If your gonna construct something, make it·as simple as·possible yet as versatile as posible.
Sapieha
What I think is that different parts of the demo which are 'packed' are overwritten in the Hub Ram when they are 'unpacked' therefore a copy of them needs to be retained in theEEProm for later use.
When they are needed they are loaded into Hub Ram from the EEProm and 'unpacked' again overwriting the Hub Ram.
A similar technique we use with PropGFX
Regards,
Coley
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
PropGFX - The home of the Hybrid Development System and PropGFX Lite
Isn't hub ram used for the VGA buffers as well?
k.
So the EEProm image may also contain stack space for the VGA framebuffer that makes it up to 32K.
I doubt that Linus would be that wasteful, instead I think different parts of the demo are unpacked from the EEProm and they are loaded into cogs and run from there leaving as much possible free ram for the display.
Remember Linus clearly states at the beginning of the demo that the Propeller has 48K Ram, this implies (to me) he is using the COG Ram to full effect.
On my demoboard P0 and P1 are toast, so I only get one of the video layers displayed.
Do an experiment pull some of your pins P0-P7 down to ground and see what effect it has, you will be enlightened.
Regards,
Coley
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
PropGFX - The home of the Hybrid Development System and PropGFX Lite
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Aka: CosmicBob
Yes I have discovered that all of the PIN 0..7 is active and in which part of the demo.
@Bob Lawrence
and watching pin 4 to 7 on my 1 channel oscilloscope was a bit boring and time-consuming (20 minutes).
Post Edited (BEEP) : 4/18/2009 8:41:43 PM GMT
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
--Steve
Propalyzer: Propeller PC Logic Analyzer
http://forums.parallax.com/showthread.php?p=788230
LOL !
ah !!! So my idea was the good one ! And so it explains/confirms the need of the demo in EEPROM (during all the time of the demo) to run. Thanks Jazzed, I feel better [noparse]:)[/noparse]
btw, I nearly finished to code my PASM disassembler. I hope to post it there tomorrow. Maybe it would help.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Composite NTSC sprite driver: http://forums.parallax.com/showthread.php?p=800114
All of the 32kb eeprom gets read in to Hub Ram at boot
though reading the first 80 bytes would be enough, you can not stop the ROMs boot loader.
Only the first 80bytes in RAM is useful, the rest is useless.
The 80 bytes is a decompression routine,
that can read the eeprom from any mem location and any byte lengths.
A demo is a bunch of small programs run in sequence, so you never need all of the code in Hub RAM at one time.
Does he use IBM Code Pack with it's 60% ratio ??
http://www.eecs.umich.edu/~tnm/compress/publications/micro32-slides.pdf
k.
1. Using the IDE, save the EEPROM as .binary
2. run the demo in EEPROM (power one so)
3. while the demo is running, using ViewPort, run the binary (so in RAM)
4. the hidden screen appear.
Don't ask me why, I found it by mistake [noparse]:)[/noparse] I don't know why ViewPort loads the stuff differently than the Prop IDE...
Excellent excellent, we slowly come all together to the same conclusions [noparse]:)[/noparse]
And yep as you said the full EEPROM is loaded by Chip's loader in RAM but I'm quite sure Linus at the beginning coded a clean loop to set the VGA buffers, scratchpad and so on.
And yes, I agree totally the demo is probably a bunch of small COGs programs packed and depacked on demand.
But, why did you say 80 bytes ? did you extract the depacker code ?
I'll take a look at the IBM stuff... but I would bet that linus has written his own packer as he did for Craft.