I just finished coding the 'send' half of xmodem in sbasic. I have xmodem in assembly, and I have xmodem coded in vb.net. But I want to be able to send programs from within an sbasic program without having to 'shell' out to CP/M - especially as the shell process is not reliable for some reason with .com files.
And the nifty thing about the propeller is that you can dump updates to the screen - something that the original assembly version can't do (it just goes blank until the file has come through).
So I can print the percentage sent on the screen.
xmodem is on terminal programs and it now exists on the propeller/cpm in various forms. It is an extremely simple and robust file transfer protocol. One wonders about a Spin version??
See attached for the source.
I've also experimented with translating sbasic to C. They are quite similar languages - eg "begin" in sbasic is { in c. Just some slight differences in the way functions are declared eg sbasic puts all the parameters in one line, where c has the parameters in subsequent lines but before the {. Sbasic also has a strange feature in that the 'main' is at the end of the program - after all the functions and procedures have been declared.
This is now transferring files between propeller boards with both wired and wireless connections.
Addit: -this is the complete wireless network program. Send a file, receive a file, log into a remote board, ask who is online etc. The xmodem send component has been improved so it builds the 131 byte packet in an array before sending it out. This is because the wireless modules don't like small delays in the middle of packets.
A>net10
Welcome to CP/M networking for the Propeller
My name is GAMMA
Network radio command list # WHO # BYE # RES # ALP # BET # GAM etc
1 - Send file to peer
2 - Receive file from peer
3 - Talk to peer
4 - WHO - search for peers
5 - RES - Reset all nearby boards
9 - BYE - Logoff
Enter your choice: 1
1=ALPHA 2=BETA 3=GAMMA 4=DELTA 5=EPSILON 6=ZETA 7=ETA 8=THETA
9=IOTA 10=KAPPA 11=LAMBDA 12=MU 13=NU 14=XI 15=OMICRON 16=PI
17=RHO 18=SIGMA 19=TAU 20=UPSILON 21=PHI 22=CHI 23=PSI 24=OMEGA
Enter board number 0 to 24: 2
Sending: #BET
Attempting login...
Now logged in to remote board BETA
Filename to send: dump.com
Please wait a moment...
Clearing port 2 buffer
Recv>XMODEM2 R DUMP.COM
XMODEM Prop port 2
Ready to receive
Waiting for an ascii 21 = ^U........................................................................................... 21Starting to send file
Number of 128 byte records = 3
1 = 33% ACK
2 = 66% ACK
3 = 100% ACK
File sent
A>net10
Welcome to CP/M networking for the Propeller
My name is GAMMA
Network radio command list # WHO # BYE # RES # ALP # BET # GAM etc
1 - Send file to peer
2 - Receive file from peer
3 - Talk to peer
4 - WHO - search for peers
5 - RES - Reset all nearby boards
9 - BYE - Logoff
Enter your choice: 3
1=ALPHA 2=BETA 3=GAMMA 4=DELTA 5=EPSILON 6=ZETA 7=ETA 8=THETA
9=IOTA 10=KAPPA 11=LAMBDA 12=MU 13=NU 14=XI 15=OMICRON 16=PI
17=RHO 18=SIGMA 19=TAU 20=UPSILON 21=PHI 22=CHI 23=PSI 24=OMEGA
Enter board number 0 to 24: 2
Sending: #BET
Attempting login...
Now logged in to remote board BETA
ESC to exit terminal program
A>
First one of my boards is working (sitting next to me running Wordstar at the moment). Brilliantly easy build did, I've done a mod to mine though as I didn't have a USB to RS232 adapter handy (I think mine's on my desk at uni but my supervisor keeps borrowing it for things), combine that with the "issues" with the compatibility of the transistor/capacitor reset circuit I decided to go with my trusty PropPlug. I soldered a 90 degree pin header in the unused 4pin row of the programmer serial port and wired off some jumps to bypass the MAX232. Photos of the build and mod are available on my site www.nathandumont.com/albums/DracBladeBuild if anyone is interested.
Thanks again to Dr_A for a well designed board.
Now to see if I can figure out how all this code works :-O
I'm adding changes to board version 6 as they get suggested. So far have added the different reset transistor and I'm thinking about audio.
I've added your proplug idea as that is fantastic and it is easy to fit on the board. See attached schematic and board layout for that corner of the board.
Re Audio - there are 4 free outputs on one of the latch chips and the ramblade driver routine can drive these. It could be possible to add an "out" port for audio and route that to one of those latch pins. Turn a pin high, turn it low. Lovely retro square wave tones...
Dr_Acula said...
Re Audio - there are 4 free outputs on one of the latch chips and the ramblade driver routine can drive these. It could be possible to add an "out" port for audio and route that to one of those latch pins. Turn a pin high, turn it low. Lovely retro square wave tones...
I think that a Spin code I/O handler will be too slow to handle audio through a Z80 port (like the TRS80 needs). The exerciser test of ini/ind/outi/outd is running for bloody ages because of the required handover of ports and values to the i/o cog - in my case.
I'd like to change I/O to use just one long at some point, as this would drastically reduce the hub RAM accesses. Instead of io_command, io_port and io_data all that info would be combined like: 31..24 = io command, 23..8 = io port (16 bits, because the Z80 uses the upper address lines for I/O also), 7..0 = io_data.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
@Dr_A: Looks like a nice solution, I was thinking of suggesting adding a PropPlug header.
Another suggestion for the version 6 board; if you placed the 3.3V linear reg "under" the 8pin dip switcher you could give a little more space to the caps/inductors and since you can never fit both at the same time you're not loosing anything. The inductors I got were a bit bigger than the footprint, combined with the fact that I only had 25V rated 470uF caps in stock it got a bit tight fitting everything in, I should have ordered some smaller ones but didn't realise until I came to build.
I've been learning a bit about how CP/M works. I've written a little python app to help me browse the disk file images on my PC, it looks a lot like a CP/M prompt but I've only implemented a couple of functions; dir and a limited pip. At the moment if you do PIP PUN:=<filename> the file will be dumped to the current working directory on your computer as a standalone file. I've used this to dump out the CBIOSX.MAC to look at the code a bit easier than on Wordstar. It's not pretty nor vastly useful, but it's taught me a lot about the filesystem which was the main aim.
Since then I've taken a chainsaw to the sdspiFemto (now called sdspiAto in my source tree) and it's losing longs like the Enterprise looses red shirts...
Seems that >50% of the code in it was for accessing i2c eeproms and booting from SD. Since neither of these are of interest at the moment I got rid of them. My plan is to get the RAM driving routines into the same cog (and maybe some other trickery, we'll see how my coding goes) and then use the old RAM driver cog to start implementing PASM I/O routines. I'll post the source once I've tidied it a bit more.
(Note the attached is a python script, not a plain text, but it gets around the mime type limitations on attachments calling it txt)
Seem to be spamming this thread at the moment, sorry.
Thought you might like to take a look at the results of my evening's work: I integrated the RAM access code into the SD card code so it's all running in a single cog now and with just under 1000 longs free on my system. Seems to be a bit slower, the RAM access is slower for the block transfers where it needs to share the pins with the SD card, I think I can optimise this but need to sleep now ( :-( ).
Generally I'm happy to have a whole free cog to play with [noparse]:D[/noparse]
hairymnstr: You really should be using the faster fsrw2.6 Unfortunately Drac chose to take an older version of ZiCog and the problem has been compounded by the later forks of the code. fsrw2.6 is superior to the older sdspiFemto code and is the result of a lot of work by lonesock and rokicki. ZiCog150 on the RamBlade and TriBlade threads contain the later fsrw2.6 (SD driver).
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔ Links to other interesting threads:
First, the good news. Your code does not seem any slower at all. But it seems slower, and the reason is that there is some code in there that sets port 2 to 1200 baud.
UART.AddPort(0,31,30,-1,-1,0,0,baud) ' set up port 0
UART.AddPort(1,25,24,-1,-1,0,0,1200) ' set up port 1 baud 1200 (called port2 through most of the code)
Now, when I booted your code up it was sending out data to port 2. This should not be happening. It only does that when it receives a magic packet via wireless that allows the board to talk to port 2.
So, why was this on?
The only place port2 is enabled is here
PRI TestPort2 | i ' look for a message eg #ALP on port 2 that opens a radio comms channel to this board
if (uart.lookbehind(1) == nickname) and (Port2ControlFlag == 0) ' look behind the head works better than looking forward from the tail. Don't have to wait 64 bytes if get out of synch
' vgatext.str(string("Open Port2"))
' uart.tx(1,"#") ' send one byte back as an acknowledge (no need for long text strings just uses up a lot of memory
uart.rxflush(1) ' clear the buffer
Port2ControlFlag := 255
Now, what you have done is removed the code that loads up the board nickname. Here in the PUB Start you had these lines commented out.
'Start the Altair I/O simulation
ReadMyName
CreateNickname
which meant that the long 'nickname' was now 0x00000000
And because the uart starts off cleared as well, it was matching 00000000 and 00000000 and turning on port2. What this meant is the bytes are outputting much slower to the screen than they need be.
How do you fix this? Well you could initialise 'nickname' as something other than zero. Or you could put the board name back in. (which just needs a tiny text file on the sd card with 8 bytes).
I put the board name back in and the display now runs a lot faster. Wordstar boots in 10 seconds which is the same as with my code.
So what I have done is put my very latest code on the first page. I see you removed the 20x4LCD code which yes will save some longs but I like the LCD. We can #ifdef that - see below. Anyway if you take that code and then integrate your combined sd card code that ought to be pretty easy. Hmm - would you name sd. something else?
If you can do that (and I think it looks pretty easy as most of the work is in your new object) then we can call this the new code. Stick your name and mods in the list at the beginning of the main.
Just for interest the new zip contains Juergens yaz80 code. Currently yaz80 does not work with the dracblade as it does not fit, and making it fit removes functionality of other things, and the aim here is to maintain and improve existing functionality. However, Juergen's code contains some incredible pasm code, including massively shrinking the sd card access and putting the 20x4 LCD into pasm. Down the track there will be some significant improvements in speed moving certain things into pasm, especially the input/output code for all the ports and Juergen has already written that code.
So - I'll stop making any mods till your code is combined.
Then - we can call that the latest version.
Then - maybe look at getting the latest fsrw code (if it is smaller or faster) as Cluso suggests
Then - see what you can do with that spare cog you have created.
Then - start to combine bits of Juergen's code one step at a time.
Re LCD - if you want to remove this
#define 20x4LCD at the beginning
search for all examples of LCD
wrap them all in #ifdef eg
LCD : "DracLCD" ' LCD 20x4 driver
and
LCD.start
and
PUB PrintStringCR(pstring)' sends pstring to the uart, to the LCD display and to the vga display
UART.str(0,pstring) ' send to the serial port
UART.str(0,string(13,10)) ' add carriage return
LCD.str(pstring) ' send to 20x4 LCD
LCD.str(string(13,10)) ' add carriage return
and
PUB PrintChar(c) ' sends c to the uart, LCD display and the vga display
UART.tx(0,c) ' send to the PC
LCD.out(c) ' send to LCD display
Re: The LCD, I started cutting out bits I didn't need with the most obvious first and since I haven't got an LCD and didn't implement the hardware latch that seemed obvious. Once I saw how few longs it saved though I don't think it's worth it, I just never bothered to uncomment those lines. I think probably we can leave it alone for now.
Re: The slow startup, I see what's going on now! I only commented out those lines while I was trying to track down a bug where accessing the SD card seemed to lock out RAM access, so I was removing "un-necessary" SD card accesses to see how far I got. Since that code is only run at boot I'm happy to just leave it in and running now the code is sorted. I'm quite interested to have a go with some of the wireless stuff myself once I'm settled into how all the code fits together properly.
I'm going to implement a couple of optimisations before I integrate the code, I thought of them last night and they should take out a couple of instruction cycles per RAM read/write which over a 128byte block will soon add up, trouble was we switched to summer time over here in the UK and I had an hour less to sleep last night :-P
@Cluso:
Thanks, I'll take a look at that. Because the code here is only really using the very low level block transfers mainly and because of the memory constraints on the DracBlade I think the only practical solution is going to be a custom re-work of the low level SPI code. So it'll depend where the speed-ups are whether they really apply where we need them.
Post Edited (hairymnstr) : 3/28/2010 12:43:42 PM GMT
Ok, I'm looking forward to seeing what you come up with.
I'm in the middle of wireless code. Can log into remote boards, send files, get files, run wordstar remotely. Working on packet routing which is getting very close now. The aim is to be able to log into any board in the network and have the data go via multiple paths using packets. So the bytes wordstar is outputting get bundled up into packets and then bounced around a mesh and finally come out the other end as if you are typing directly in the program.
Thanks, I'll take a look at that. Because the code here is only really using the very low level block transfers mainly and because of the memory constraints on the DracBlade I think the only practical solution is going to be a custom re-work of the low level SPI code. So it'll depend where the speed-ups are whether they really apply where we need them.
Be careful! I don't yet know what it is, but trying to use the mb_rawb_spi26_rr006.spin instead of sdspiFemto.spin is not working (right) on my side. I have code to verify that the image files are sequential. It follows the cluster chain in the sector buffer and this fails at varying points, though I'm absolutely sure the files are in order - they all verified okay with the sdspiFemto.spin and also by looking at a local image of the SD card. There is something fishy going on, I just don't yet know where exactly.
Another funny side-effect I realized just now: When I disable my new tv80 driver, initializing the SD card seems to hang.
Anway, I just wanted to say that if you experience strange things when trying this code, you have been warned [noparse]:)[/noparse]
Juergen
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
@pullmoll, the only link between the keyboard and SD I can think of is if you're using Dr_A's trick of putting the SD buffer in the space that the keyboard driver occupied in hub, but I guess you've altered all this for your current setup?
hairymnstr said...
@pullmoll, the only link between the keyboard and SD I can think of is if you're using Dr_A's trick of putting the SD buffer in the space that the keyboard driver occupied in hub, but I guess you've altered all this for your current setup?
I'm using a VAR iobuff. It looks like I get call stack corruptions from somewhere, i.e. a function doesn't return the expected value, but something different.
This could actually be anything. I'm trying to identify the bad guy. However, it all started to get worse when I tried to use the new SPI functions.
Edit: Okay, I take that back. After doing a test with just the mb_rawb_spi26_rr006, Keyboard.spin, cserial.spin and my fatfs.spin the verification of cluster chains works, so it has to be one of the remaining modules that goes haywire. Corrected correction: It does go wrong. Now I'll remove one after another to find the candidate.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
re fsrw2.6: This is the low level driver and uses counters to clock the SD data in and out faster. IIRC the code is smaller but there is a buffer within the cog.
NOTE: The driving method is subtly different to the older sdspifemto driver. This could be a source of problems. I have added a couple of additional external methods to get the current SD sector, etc.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔ Links to other interesting threads:
I've modified your latest set of spinfiles. It seems to be all working on my system (CP/M boots and I can dirx on A: and B[noparse]:)[/noparse] I've not tested any more than that though.
I've had a look at fsrw2.6 it has a lot of the "missing" features that we need (HCSD support (because I can't get non-hcsd cheap any more [noparse]:([/noparse] ), and therefore FAT32 support etc). It's going to take a while to get my head around how it works. The important question is whether the PASM can be squashed enough to share the cog with the RAM drivers. I think I can see some bits I can "trim" as there are some block transfer routines for transferring more than 512 bytes at a time and there simply isn't room for that on the DracBlade so they can go. I'll see what can be done.
I'm hoping to fit the 512byte buffer into the cog RAM so that the CP/M "DMA" requests can be serviced without transfers to/from hub. Whether that is possible remains to be seen...
EDIT:
Dr_A: Can you explain the reasoning behind resetting the SD card periodically regardless of whether an error has occurred? I don't see that there's any point to this, and it won't work well with the new FSRW code.
I think I can fit the RAM code in if we use the mb_small_spi.spin. It's a very clever bit of code and has the 512byte buffer already declared in the cog so things are looking good.
Pullmoll: Is your var buffer long aligned? I noticed the new fsrw wants long aligned space, if it's declared as byte and you're including and removing objects it's possible the buffer is moving to non-long aligned offsets when you compile different combinations of code.
Post Edited (hairymnstr) : 3/28/2010 3:16:19 PM GMT
hairymnstr said...
Pullmoll: Is your var buffer long aligned? I noticed the new fsrw wants long aligned space, if it's declared as byte and you're including and removing objects it's possible the buffer is moving to non-long aligned offsets when you compile different combinations of code.
Yes, it is. This is not the cause of my trouble. I found the miserable bum who's going nuts in memory! Guess what? It's Dracblade.spin. I can verify this in a test program that has nothing to do with yaz80, but just the other objects I use. Depending on where I move the initialization of xmm (which is the object name for the various XMM handlers I chose) various strange effects happen. When I move it after the serial object, the terminal spits out trash. As soon as I disable it entirely, everything goes fine.
Is it possible that Dracblade.spin modifies output pins which it shouldn't? Perhaps that could explain the strange effects.
No, it was me idiot :-/ Argh.. this bug is going to kill me!
Edit: The SPI object is much faster than sdspiFemto. I'd guess by a factor of 5-10.
Edit2: Latest observation is, it looks like the SPI driver wants to run on cog 1 or 2, not on cog 3 or higher. If I load any objects into 1+2, then it seems to fail.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Re "Dr_A: Can you explain the reasoning behind resetting the SD card periodically"
You mean the way it disables the sd card after 3 seconds of not being used? That is to conserve power when battery powered. It is the same as some floppy drives that used to stop spinning after a while. Also I figured that if someone pulled the card out at a random time it was less likely to corrupt the data if the card was disabled most of the time. Maybe that logic is flawed?
Re combining things into one cog - that sounds great.
Assuming I can get past the bugs that pullmoll has been seeing; I have been studying the new block transfer code in FSRW and it is ingenious. I think with the mb_small_spi.spin I can fit some extra code into the cog. What I hope to do is implement a 128byte optimised block transfer from SD to external RAM allowing (and back) making the 128 byte "records" from CP/M to make best use of the driver. I think I can see how this could be implemented with the new FSRW code, but it is going to be tight on cog space.
The problem with re-setting the SD is that it takes _ages_ hundreds of milliseconds at the least, and this seriously impacts on performance. I agree that un-mounting the device regularly protects it, but at what cost? Once mounted the FSRW2.6 code assumes the card remains constant until the cog is stopped or a halt is called on the device. This is quite reasonable, if you go unplugging cards at random you deserve all the trouble you get IMO. Generally though, the new FSRW does the necessary card sleep etc. when suitable, I think having the higher layer periodically restart it is only going to cause problems later on. I'll start trying some things with my new code and see how it goes.
Well maybe the sleep is not needed if the new code has sleep anyway? I guess I should get a multimeter out and measure the current draw. There are quotes on websites saying over a hundred ma but I don't think it does draw that much because it doesn't get warm (the only warm chip is the propeller). So ok, lets just go with that new code and ignore the sleep and you can then delete some more longs in the main program as well - eg the counter and disable code (maybe just comment them out rather than delete them).
Cluso is our resident expert on 128 vs 512 bytes. There is something complicated there that I don't understand.
As an aside what exciting things have you got planned for the spare cog?
1)Audio?
2) I've been intrigued by the bell modem code which allows you to send data from a speaker to a microphone, and/or via a CB radio.
3) Infrared comms?
The 128/512 byte issue (as I see it, others undoubtedly have more experience, I only booted CP/M for the first time a day and a half ago) is that SD cards (and other modern media e.g. HDD and USB drives) use 512byte logical blocks (generally anyway) so reading from an SD provides a block of 512 bytes. However CP/M generally seems to be asking for 128 byte disk sectors. Based on the way the current code is working I think that each 512 byte SD sector directly maps to 4 128 byte CP/M sectors (or records depending on who's documentation you read). So if I take the spi code that pre-buffers one 512 byte sector ahead and adjust it so that it buffers only after 4 128 byte blocks have been requested we should see significant improvements.
I may be barking up the wrong tree here but I think the key is keeping track of which 128 byte CP/M "sector/record" we're looking at.
As far as the new cog is concerned I was thinking considerably more mundane; along the lines of faster I/O handling. Although this would benefit any attempts at networking etc.
Yes if you fetch 512 bytes off an sd card and cp/m is in the middle of reading a file there is a good chance that those four 128 byte blocks will be contiguous.
Writing to the disk is a little more complex. cp/m writes 128 bytes. Do you then write them immediately to the sd card or do you wait a little bit because there is a fairly good chance the next 128 will be contiguous? And if you do wait - do you eventually write those 128 if no further writes are requested - which implies a little timer/counter is needed.
Re speed, the 4 in one serial driver code only has 64 byte buffers. xmodem needs 131 bytes. So xmodem needs to be gobbling up the bytes faster than they are arriving. That means the sd card code needs to be fast. At the moment the fastest speed xmodem can work is 19200 baud. Faster sd card will speed this up. Another option is one cog for each serial port and that will mean bigger buffers and then it can gobble up an entire xmodem packet in one go. Though that may not make a transfer any faster - you can send the data at 1megabit but if it takes a certain amount of time to write a packet to sd card then xmodem is not going to send an ACK any faster.
I did all the N8VEM coding at 38k and even getting the speed from 19.2 to 38.4k would make development of programs a lot faster. So any improvements to the sd code are well worth doing.
I was planning on having a "dirty" bit that causes the write to be done when a (read) or (write outside the current 512byte block) is performed. (Parenthesis added for clarity of intentions).
Re: xmodem, what layer is this running in? Is it a CP/M program or is it running in spin or pasm in another cog? This could affect the way disk access is optimised.
hairymnstr: The disk emulation code in ZiCog holds a 512 byte SD block in a buffer. For reads of 128 byte CP/M sectors it checks if that sector exists within the currently loaded 512 byte block and if so just returns the appropriate part of the block. If not a new block is read from SD. So generally there is only one SD block read for 4 CP/M sector reads. There is no need to make any changes to the SPI code. It all works as you want already.
For 128 byte CP/M sector writes the correct area of the block is updated and then the whole block is written out to SD. This means one SD block write for each CP/M sector write.
We could speed this up by delaying writes until a write to a non-loaded sector is made. This was not done yet as we worried about disk integrity. Can we be sure that last block gets flushed to SD after the last sector write CP/M makes.
Thinking about it now, CP/M works in 2K byte clusters so as far as I can tell it will always write out a minimum of 2K of data. This means that by keeping track of the amount of data written on could always flush the block when 2K has been written thus guaranteeing the"delayed" write is always made and disk integrity is ensured.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
heater said...
For 128 byte CP/M sector writes the correct area of the block is updated and then the whole block is written out to SD. This means one SD block write for each CP/M sector write.
We could speed this up by delaying writes until a write to a non-loaded sector is made. This was not done yet as we worried about disk integrity. Can we be sure that last block gets flushed to SD after the last sector write CP/M makes.
FWIW I have a timed flush in io.spin. It remembers CNT on every write and the loop waiting for a command checks if the timeout (1 second) is reached. If it is, the 512 byte block is written. I don't think anyone would e.g. compile a big file and then pull out the SD card, and if he does, he rightfully ought to be in trouble.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Thanks for the clarification heater. This is how I understood the code to be working. The reason I'm trying to re-invent the wheel, as it were, is that when I moved the RAM access functions for the DracBlade into the SD cog I saw an opportunity to move some of this logic into the PASM and avoid buffering the data in and out of HUB. This is going to require some amendment to the new FSRW code because of the way it does automatic read-ahead, which we don't want until CP/M has loaded all the 128 byte blocks from that SD sector.
I hadn't spotted the issue of delayed write pending on shutdown that is an interesting problem. If the writes can be guaranteed to finish on a 512byte (as they will for 2K transfers) block boundary then it could be coded to flush when the last of the four 128byte blocks was written.
Is it 2K or 4K? The dpb indicates 4K "tracks" if I remember correctly.
2K or 4K? I'm not sure now. I would not worry about what's in the dpb for tracks. I'm pretty sure all file sizes I've seen in CP/M are multiples of 2K. In fact CP/M has no way to express a file size to a higher resolution than that.
Anyway, what does it matter? I'm fairly certain that if you flushed the buffer after 4 sector 128 byte sector writes that would be safe.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
Re "Re: xmodem, what layer is this running in? Is it a CP/M program or is it running in spin or pasm in another cog?"
xmodem is running in cp/m but it is using direct port I/O calls rather than bios calls. So any improvements to OUT and IN in the emulation will speed this up (ie less spin code, more pasm which Juergen is working on).
So xmodem collects 128 bytes then writes them to an open file.
Re "If the writes can be guaranteed to finish on a 512byte..." that might work for sequential files but in the CP/M bios there are sequential files and random files, and random files are going to be more complicated as you have to be sure to flush the buffer as the last 128 byte record could be anywhere. I think Juergen has the right idea with an automatic flush after 1 second.
I'm pretty sure the minimum file size is 2k - ie write 1 byte to a new file and close it and do a STAT on that file and the size will be 2k.
I think Juergen is looking a complete new i/o routine. If so there can be some big improvements - eg I poll every second or so looking for magic packets on port2 to do wireless things - but it depends on OUT and IN calls for the counter. So if a program has no OUT or IN calls eg a big maths calculation it will never check port 2. Counters in pasm would be a lot neater.
And the VT100 in pasm? Well that would be all my Christmas presents all at once!
Dr_A: The CP/M BIOS makes no distinction between sequential and random files. It only provides the mechanism to the BDOS to read and write sectors.
Either way you have a point. I am pinning my argument on the fact that CP/M is working in blocks of 2K. Seeing as you always end up with a 2K file even if you only write one byte to it.
BUT any program that writes sectors directly would become unreliable if only sector counting were used. The idea of timing out the writes is the way to go and was thought about a long time ago. Glad to see Juergen is on the case here.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
Dr_Acula said...
And the VT100 in pasm? Well that would be all my Christmas presents all at once!
Way too big. I tried it and the cog ran full after writing abt. 1/3 of only the basic commands. Then I tried LMM, but that doesn't work as a task besides the video generation code. In a dedicated cog a VT100 PASM implementation would be possible, but actually I don't see toomuch speed gain there. All scrolling, clearing, line deletions and insertions can be done with bytemove/longmove and that's as fast as it can get.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Comments
And the nifty thing about the propeller is that you can dump updates to the screen - something that the original assembly version can't do (it just goes blank until the file has come through).
So I can print the percentage sent on the screen.
xmodem is on terminal programs and it now exists on the propeller/cpm in various forms. It is an extremely simple and robust file transfer protocol. One wonders about a Spin version??
See attached for the source.
I've also experimented with translating sbasic to C. They are quite similar languages - eg "begin" in sbasic is { in c. Just some slight differences in the way functions are declared eg sbasic puts all the parameters in one line, where c has the parameters in subsequent lines but before the {. Sbasic also has a strange feature in that the 'main' is at the end of the program - after all the functions and procedures have been declared.
This is now transferring files between propeller boards with both wired and wireless connections.
Addit: -this is the complete wireless network program. Send a file, receive a file, log into a remote board, ask who is online etc. The xmodem send component has been improved so it builds the 131 byte packet in an array before sending it out. This is because the wireless modules don't like small delays in the middle of packets.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Post Edited (Dr_Acula) : 3/20/2010 1:55:46 AM GMT
Thanks again to Dr_A for a well designed board.
Now to see if I can figure out how all this code works :-O
I'm adding changes to board version 6 as they get suggested. So far have added the different reset transistor and I'm thinking about audio.
I've added your proplug idea as that is fantastic and it is easy to fit on the board. See attached schematic and board layout for that corner of the board.
Re Audio - there are 4 free outputs on one of the latch chips and the ramblade driver routine can drive these. It could be possible to add an "out" port for audio and route that to one of those latch pins. Turn a pin high, turn it low. Lovely retro square wave tones...
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Post Edited (Dr_Acula) : 3/27/2010 12:44:21 AM GMT
I think that a Spin code I/O handler will be too slow to handle audio through a Z80 port (like the TRS80 needs). The exerciser test of ini/ind/outi/outd is running for bloody ages because of the required handover of ports and values to the i/o cog - in my case.
I'd like to change I/O to use just one long at some point, as this would drastically reduce the hub RAM accesses. Instead of io_command, io_port and io_data all that info would be combined like: 31..24 = io command, 23..8 = io port (16 bits, because the Z80 uses the upper address lines for I/O also), 7..0 = io_data.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Another suggestion for the version 6 board; if you placed the 3.3V linear reg "under" the 8pin dip switcher you could give a little more space to the caps/inductors and since you can never fit both at the same time you're not loosing anything. The inductors I got were a bit bigger than the footprint, combined with the fact that I only had 25V rated 470uF caps in stock it got a bit tight fitting everything in, I should have ordered some smaller ones but didn't realise until I came to build.
Since then I've taken a chainsaw to the sdspiFemto (now called sdspiAto in my source tree) and it's losing longs like the Enterprise looses red shirts...
Seems that >50% of the code in it was for accessing i2c eeproms and booting from SD. Since neither of these are of interest at the moment I got rid of them. My plan is to get the RAM driving routines into the same cog (and maybe some other trickery, we'll see how my coding goes) and then use the old RAM driver cog to start implementing PASM I/O routines. I'll post the source once I've tidied it a bit more.
(Note the attached is a python script, not a plain text, but it gets around the mime type limitations on attachments calling it txt)
Thought you might like to take a look at the results of my evening's work: I integrated the RAM access code into the SD card code so it's all running in a single cog now and with just under 1000 longs free on my system. Seems to be a bit slower, the RAM access is slower for the block transfers where it needs to share the pins with the SD card, I think I can optimise this but need to sleep now ( :-( ).
Generally I'm happy to have a whole free cog to play with [noparse]:D[/noparse]
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)·
· Prop OS: SphinxOS·, PropDos , PropCmd··· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBlade Props: www.cluso.bluemagic.biz
Ok, lets work together on this.
First, the good news. Your code does not seem any slower at all. But it seems slower, and the reason is that there is some code in there that sets port 2 to 1200 baud.
Now, when I booted your code up it was sending out data to port 2. This should not be happening. It only does that when it receives a magic packet via wireless that allows the board to talk to port 2.
So, why was this on?
The only place port2 is enabled is here
Now, what you have done is removed the code that loads up the board nickname. Here in the PUB Start you had these lines commented out.
which meant that the long 'nickname' was now 0x00000000
And because the uart starts off cleared as well, it was matching 00000000 and 00000000 and turning on port2. What this meant is the bytes are outputting much slower to the screen than they need be.
How do you fix this? Well you could initialise 'nickname' as something other than zero. Or you could put the board name back in. (which just needs a tiny text file on the sd card with 8 bytes).
I put the board name back in and the display now runs a lot faster. Wordstar boots in 10 seconds which is the same as with my code.
So what I have done is put my very latest code on the first page. I see you removed the 20x4LCD code which yes will save some longs but I like the LCD. We can #ifdef that - see below. Anyway if you take that code and then integrate your combined sd card code that ought to be pretty easy. Hmm - would you name sd. something else?
If you can do that (and I think it looks pretty easy as most of the work is in your new object) then we can call this the new code. Stick your name and mods in the list at the beginning of the main.
Just for interest the new zip contains Juergens yaz80 code. Currently yaz80 does not work with the dracblade as it does not fit, and making it fit removes functionality of other things, and the aim here is to maintain and improve existing functionality. However, Juergen's code contains some incredible pasm code, including massively shrinking the sd card access and putting the 20x4 LCD into pasm. Down the track there will be some significant improvements in speed moving certain things into pasm, especially the input/output code for all the ports and Juergen has already written that code.
So - I'll stop making any mods till your code is combined.
Then - we can call that the latest version.
Then - maybe look at getting the latest fsrw code (if it is smaller or faster) as Cluso suggests
Then - see what you can do with that spare cog you have created.
Then - start to combine bits of Juergen's code one step at a time.
Re LCD - if you want to remove this
#define 20x4LCD at the beginning
search for all examples of LCD
wrap them all in #ifdef eg
and
and
and
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Post Edited (Dr_Acula) : 3/28/2010 11:22:39 AM GMT
Re: The LCD, I started cutting out bits I didn't need with the most obvious first and since I haven't got an LCD and didn't implement the hardware latch that seemed obvious. Once I saw how few longs it saved though I don't think it's worth it, I just never bothered to uncomment those lines. I think probably we can leave it alone for now.
Re: The slow startup, I see what's going on now! I only commented out those lines while I was trying to track down a bug where accessing the SD card seemed to lock out RAM access, so I was removing "un-necessary" SD card accesses to see how far I got. Since that code is only run at boot I'm happy to just leave it in and running now the code is sorted. I'm quite interested to have a go with some of the wireless stuff myself once I'm settled into how all the code fits together properly.
I'm going to implement a couple of optimisations before I integrate the code, I thought of them last night and they should take out a couple of instruction cycles per RAM read/write which over a 128byte block will soon add up, trouble was we switched to summer time over here in the UK and I had an hour less to sleep last night :-P
@Cluso:
Thanks, I'll take a look at that. Because the code here is only really using the very low level block transfers mainly and because of the memory constraints on the DracBlade I think the only practical solution is going to be a custom re-work of the low level SPI code. So it'll depend where the speed-ups are whether they really apply where we need them.
Post Edited (hairymnstr) : 3/28/2010 12:43:42 PM GMT
I'm in the middle of wireless code. Can log into remote boards, send files, get files, run wordstar remotely. Working on packet routing which is getting very close now. The aim is to be able to log into any board in the network and have the data go via multiple paths using packets. So the bytes wordstar is outputting get bundled up into packets and then bounced around a mesh and finally come out the other end as if you are typing directly in the program.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Be careful! I don't yet know what it is, but trying to use the mb_rawb_spi26_rr006.spin instead of sdspiFemto.spin is not working (right) on my side. I have code to verify that the image files are sequential. It follows the cluster chain in the sector buffer and this fails at varying points, though I'm absolutely sure the files are in order - they all verified okay with the sdspiFemto.spin and also by looking at a local image of the SD card. There is something fishy going on, I just don't yet know where exactly.
Another funny side-effect I realized just now: When I disable my new tv80 driver, initializing the SD card seems to hang.
Anway, I just wanted to say that if you experience strange things when trying this code, you have been warned [noparse]:)[/noparse]
Juergen
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Post Edited (pullmoll) : 3/28/2010 1:05:10 PM GMT
I'm using a VAR iobuff. It looks like I get call stack corruptions from somewhere, i.e. a function doesn't return the expected value, but something different.
This could actually be anything. I'm trying to identify the bad guy. However, it all started to get worse when I tried to use the new SPI functions.
Edit: Okay, I take that back. After doing a test with just the mb_rawb_spi26_rr006, Keyboard.spin, cserial.spin and my fatfs.spin the verification of cluster chains works, so it has to be one of the remaining modules that goes haywire. Corrected correction: It does go wrong. Now I'll remove one after another to find the candidate.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Post Edited (pullmoll) : 3/28/2010 2:13:25 PM GMT
NOTE: The driving method is subtly different to the older sdspifemto driver. This could be a source of problems. I have added a couple of additional external methods to get the current SD sector, etc.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:
· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)·
· Prop OS: SphinxOS·, PropDos , PropCmd··· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBlade Props: www.cluso.bluemagic.biz
I've had a look at fsrw2.6 it has a lot of the "missing" features that we need (HCSD support (because I can't get non-hcsd cheap any more [noparse]:([/noparse] ), and therefore FAT32 support etc). It's going to take a while to get my head around how it works. The important question is whether the PASM can be squashed enough to share the cog with the RAM drivers. I think I can see some bits I can "trim" as there are some block transfer routines for transferring more than 512 bytes at a time and there simply isn't room for that on the DracBlade so they can go. I'll see what can be done.
I'm hoping to fit the 512byte buffer into the cog RAM so that the CP/M "DMA" requests can be serviced without transfers to/from hub. Whether that is possible remains to be seen...
EDIT:
Dr_A: Can you explain the reasoning behind resetting the SD card periodically regardless of whether an error has occurred? I don't see that there's any point to this, and it won't work well with the new FSRW code.
I think I can fit the RAM code in if we use the mb_small_spi.spin. It's a very clever bit of code and has the 512byte buffer already declared in the cog so things are looking good.
Pullmoll: Is your var buffer long aligned? I noticed the new fsrw wants long aligned space, if it's declared as byte and you're including and removing objects it's possible the buffer is moving to non-long aligned offsets when you compile different combinations of code.
Post Edited (hairymnstr) : 3/28/2010 3:16:19 PM GMT
Yes, it is. This is not the cause of my trouble.
I found the miserable bum who's going nuts in memory! Guess what? It's Dracblade.spin. I can verify this in a test program that has nothing to do with yaz80, but just the other objects I use. Depending on where I move the initialization of xmm (which is the object name for the various XMM handlers I chose) various strange effects happen. When I move it after the serial object, the terminal spits out trash. As soon as I disable it entirely, everything goes fine.
Is it possible that Dracblade.spin modifies output pins which it shouldn't? Perhaps that could explain the strange effects.
No, it was me idiot :-/ Argh.. this bug is going to kill me!
Edit: The SPI object is much faster than sdspiFemto. I'd guess by a factor of 5-10.
Edit2: Latest observation is, it looks like the SPI driver wants to run on cog 1 or 2, not on cog 3 or higher. If I load any objects into 1+2, then it seems to fail.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
Post Edited (pullmoll) : 3/28/2010 7:12:06 PM GMT
You mean the way it disables the sd card after 3 seconds of not being used? That is to conserve power when battery powered. It is the same as some floppy drives that used to stop spinning after a while. Also I figured that if someone pulled the card out at a random time it was less likely to corrupt the data if the card was disabled most of the time. Maybe that logic is flawed?
Re combining things into one cog - that sounds great.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
The problem with re-setting the SD is that it takes _ages_ hundreds of milliseconds at the least, and this seriously impacts on performance. I agree that un-mounting the device regularly protects it, but at what cost? Once mounted the FSRW2.6 code assumes the card remains constant until the cog is stopped or a halt is called on the device. This is quite reasonable, if you go unplugging cards at random you deserve all the trouble you get IMO. Generally though, the new FSRW does the necessary card sleep etc. when suitable, I think having the higher layer periodically restart it is only going to cause problems later on. I'll start trying some things with my new code and see how it goes.
Cluso is our resident expert on 128 vs 512 bytes. There is something complicated there that I don't understand.
As an aside what exciting things have you got planned for the spare cog?
1)Audio?
2) I've been intrigued by the bell modem code which allows you to send data from a speaker to a microphone, and/or via a CB radio.
3) Infrared comms?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Post Edited (Dr_Acula) : 3/28/2010 11:00:58 PM GMT
I may be barking up the wrong tree here but I think the key is keeping track of which 128 byte CP/M "sector/record" we're looking at.
As far as the new cog is concerned I was thinking considerably more mundane; along the lines of faster I/O handling. Although this would benefit any attempts at networking etc.
Writing to the disk is a little more complex. cp/m writes 128 bytes. Do you then write them immediately to the sd card or do you wait a little bit because there is a fairly good chance the next 128 will be contiguous? And if you do wait - do you eventually write those 128 if no further writes are requested - which implies a little timer/counter is needed.
Re speed, the 4 in one serial driver code only has 64 byte buffers. xmodem needs 131 bytes. So xmodem needs to be gobbling up the bytes faster than they are arriving. That means the sd card code needs to be fast. At the moment the fastest speed xmodem can work is 19200 baud. Faster sd card will speed this up. Another option is one cog for each serial port and that will mean bigger buffers and then it can gobble up an entire xmodem packet in one go. Though that may not make a transfer any faster - you can send the data at 1megabit but if it takes a certain amount of time to write a packet to sd card then xmodem is not going to send an ACK any faster.
I did all the N8VEM coding at 38k and even getting the speed from 19.2 to 38.4k would make development of programs a lot faster. So any improvements to the sd code are well worth doing.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Re: xmodem, what layer is this running in? Is it a CP/M program or is it running in spin or pasm in another cog? This could affect the way disk access is optimised.
For 128 byte CP/M sector writes the correct area of the block is updated and then the whole block is written out to SD. This means one SD block write for each CP/M sector write.
We could speed this up by delaying writes until a write to a non-loaded sector is made. This was not done yet as we worried about disk integrity. Can we be sure that last block gets flushed to SD after the last sector write CP/M makes.
Thinking about it now, CP/M works in 2K byte clusters so as far as I can tell it will always write out a minimum of 2K of data. This means that by keeping track of the amount of data written on could always flush the block when 2K has been written thus guaranteeing the"delayed" write is always made and disk integrity is ensured.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
FWIW I have a timed flush in io.spin. It remembers CNT on every write and the loop waiting for a command checks if the timeout (1 second) is reached. If it is, the 512 byte block is written. I don't think anyone would e.g. compile a big file and then pull out the SD card, and if he does, he rightfully ought to be in trouble.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.
I hadn't spotted the issue of delayed write pending on shutdown that is an interesting problem. If the writes can be guaranteed to finish on a 512byte (as they will for 2K transfers) block boundary then it could be coded to flush when the last of the four 128byte blocks was written.
Is it 2K or 4K? The dpb indicates 4K "tracks" if I remember correctly.
Anyway, what does it matter? I'm fairly certain that if you flushed the buffer after 4 sector 128 byte sector writes that would be safe.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
xmodem is running in cp/m but it is using direct port I/O calls rather than bios calls. So any improvements to OUT and IN in the emulation will speed this up (ie less spin code, more pasm which Juergen is working on).
So xmodem collects 128 bytes then writes them to an open file.
Re "If the writes can be guaranteed to finish on a 512byte..." that might work for sequential files but in the CP/M bios there are sequential files and random files, and random files are going to be more complicated as you have to be sure to flush the buffer as the last 128 byte record could be anywhere. I think Juergen has the right idea with an automatic flush after 1 second.
I'm pretty sure the minimum file size is 2k - ie write 1 byte to a new file and close it and do a STAT on that file and the size will be 2k.
I think Juergen is looking a complete new i/o routine. If so there can be some big improvements - eg I poll every second or so looking for magic packets on port2 to do wireless things - but it depends on OUT and IN calls for the counter. So if a program has no OUT or IN calls eg a big maths calculation it will never check port 2. Counters in pasm would be a lot neater.
And the VT100 in pasm? Well that would be all my Christmas presents all at once!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller
Post Edited (Dr_Acula) : 3/29/2010 11:18:22 AM GMT
Either way you have a point. I am pinning my argument on the fact that CP/M is working in blocks of 2K. Seeing as you always end up with a 2K file even if you only write one byte to it.
BUT any program that writes sectors directly would become unreliable if only sector counting were used. The idea of timing out the writes is the way to go and was thought about a long time ago. Glad to see Juergen is on the case here.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
For me, the past is not over yet.
Way too big. I tried it and the cog ran full after writing abt. 1/3 of only the basic commands. Then I tried LMM, but that doesn't work as a task besides the video generation code. In a dedicated cog a VT100 PASM implementation would be possible, but actually I don't see toomuch speed gain there. All scrolling, clearing, line deletions and insertions can be done with bytemove/longmove and that's as fast as it can get.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
He died at the console of hunger and thirst.
Next day he was buried. Face down, nine edge first.