YModem (2019 update)

Rayman · 2008-03-27 19:42

Here's a program you can load (ymodem.spin) that will let you transfer files from an SD card attached to a Propeller over serial to a PC. I've tested it using ExtraPutty on Win10 with Activity Board WX.

It's not really optimized, but still achieves ~1.5kB/s transfer rates at 115200 baud.
You can also view the SD card root directory and erase files using HyperTerminal or ExtraPutty.
Note that you don't have too long to select a file to send or save to location before it times out...
This can be useful in cases where it is inconvenient to remove the SD card from the Propeller...
Ymodem is better than Xmodem because you don't have to say the filename twice and the filesize comes out excactly right.· (but zmodem is better...)

The YModem2 is the 2019 update. Updated FSRW to 2.6 and change default pins to match Activity Board WX.
Note: cheezus also posted a new version, look down a few posts here.

stevenmess2004 · 2008-03-27 19:55

Rayman said...
(but zmodem is better...)

So the question then becomes, where do I get zmodem

or do I have to write it

Rayman · 2008-03-27 20:00

I think zmodem is useful for comms over telephone lines where there is a high likelyhood of errors and failures. However, in this application the error rate is extremely low, so I don't think it'd be worth the effort. I even think CRC is overkill. I just implemented it for compatibility with Hyperterminal...

I just sent a 2MB file the Prop without a single bit error...

Oldbitcollector (Jeff) · 2008-03-27 21:17

Very Nice!

OBC

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
New to the Propeller?

Getting started with the Protoboard? - Propeller Cookbook 1.4
Updates to the Cookbook are now posted to: Propeller.warrantyvoid.us
Got an SD card? - PropDOS
A Living Propeller FAQ - The Propeller Wiki
(Got the Knowledge? Got a Moment? Add something today!)

parts-man73 · 2008-03-27 22:07

Jeff, Think that might fit into PropComm ? Then you could not only connect to BBS's, but download files.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Brian

uController.com - home of SpinStudio - the modular Development system for the Propeller

PropNIC - Add ethernet ability to your Propeller! PropJoy - Plug in a joystick and play some games!

SD card Adapter - mass storage for the masses Audio/Video adapter add composite video and sound to your Proto Board

Javalin · 2008-03-28 12:15

hey thats cool - thanks for sharing...!

J

Rayman · 2008-03-28 20:20

I think this is also going to provide an easy way to transmit data, saved in HUB RAM or EEPROM, to the PC as a file... I've modified the code to do this and hope to test it out next week.

groggory · 2008-03-31 19:46

Originally, I was going to be using an ALFATxp (java on chip) to deal with my SD card but have since moved to a completely propeller based project design. In my java design I purchased a kermit library for sending files but no such thing is available for propeller.

Is this alpha YModem code the best out there so far? Or do I have to go and write it up myself? What are your recommendations for sending files over serial? Note, I am sending data via serial on an XBee, so there might be occasional packet loss. That is why I need a protocol that checks for that sort of thing.

Thanks..

Greg

Rayman · 2010-04-02 23:44

Weird... Why don't I see groggory's above post in the Prop forum?

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Rayman · 2010-04-02 23:46

Now I see it [noparse]:)[/noparse]

groggory: I think it's the best [noparse]:)[/noparse]
I have upgraded it a bit to send data from RAM as a file too, not just SD card files. I'll try to find time to post my latest version. It's pretty handy for datalogging applications...

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Rayman · 2010-04-02 23:49

Strange... groggory did a stealth post somehow... If you click on his name, you don't see his post in this thread...
I'm also pretty sure his post didn't bump this thread... Wierd!

Is there a glitch in the matrix?

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Dave Hein · 2010-04-22 18:02

Rayman,

I tried your code with an EEPROM file system I'm writing, and it works great transferring data from the Prop to the PC.· However, I'm having problems transferring·files the other way.· I'm running at a baudrate of 57,600, and it gets stuck receiving the first 128-byte message.· I suspect that it is·unable to keep up at·57,600 and it is losing bytes.· How fast have you run it on a PC-to-Prop transfer?

Dave

Rayman · 2010-04-22 18:12

I seem to recall running at 115200, but the code I actually use it in only uses 9600 baud for some reason.
My top post quotes speeds for 115200, so I guess I tested it at that speed.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Dave Hein · 2010-04-22 19:31

I'll try it at different speeds tonight when I get home.· I suspect the Spin code on the receive tops out at a lower rate because of the CRC calculation.· I have implemented a Spin serial output in the past that can only run up to 19,200.· The CRC computation requires a similar amount of compute as a serial output routine.

Dave

Rayman · 2010-04-22 20:54

I don't think that's the problem, but it's been a while since I've tested it...
Are you using Hyperterminal? I've only tested it with Hyperterminal. Some other program may have different timeouts or something...

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Cluso99 · 2010-04-22 23:56

@Rayman: This will be a good piece of code to add to SphinxOS

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Links to other interesting threads:

· Home of the MultiBladeProps: TriBlade,·RamBlade,·SixBlade, website
· Single Board Computer:·3 Propeller ICs·and a·TriBladeProp board (ZiCog Z80 Emulator)
· Prop Tools under Development or Completed (Index)
· Emulators: CPUs Z80 etc; Micros Altair etc;· Terminals·VT100 etc; (Index) ZiCog (Z80) , MoCog (6809)·
· Prop OS: SphinxOS·, PropDos , PropCmd··· Search the Propeller forums·(uses advanced Google search)
My cruising website is: ·www.bluemagic.biz·· MultiBlade Props: www.cluso.bluemagic.biz

Dave Hein · 2010-04-23 04:33

Rayman,

I got the PC-to-Prop transfer working.· I was able to download a 13Kbyte file from my PC to the EEProm file system on the Prop.· The main problem was that my serial receive buffer was too small, and it was wrapping around.· I noticed that the receive loop for 1024-byte packets writes directly into the buffer, and then does the CRC computation after receiving the packet.· The receive loops·for the first 128-byte packets and subsequent 128-byte packets computes the CRC·within the loop,·which slows down the processing.

I was able to run at 57,600 by using a very large receive buffer in the serial driver.· Even with the tight loop for the 1024-byte receive code, it still contains a call to ser.rx in each loop.· A lot of Spin code is executed in each call, so I don't think it would keep up with the 57,600 baud rate without a large receive buffer.· I'll do some more timing tests to see what the maximum baud rate is for a small buffer.

Thanks for providing the Ymodem code.· It will be very usefull for sending files back-and-forth between the Prop and the PC.

Dave

·

Dr_Acula · 2010-04-23 06:02

Re Dave "I have implemented a Spin serial output in the past that can only run up to 19,200." That is interesting, because using completely different code (CP/M, xmodem) I find the max speed is also 19200. I think it might be limited by the write speed to the sd card, and given this is working in well optimised pasm, maybe it is hitting a fundamental limit? Interestingly, with xmodem packets (131 bytes) and 256 byte buffers, it is possible to run at 115200 baud, and possibly even more. However, the overall data transfer rate does not get faster than something under about 38400, as those bytes are sitting in the buffer and still have to be processed.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
www.smarthome.viviti.com/propeller

Post Edited (Dr_Acula) : 4/23/2010 6:13:42 AM GMT

Rayman · 2010-04-23 12:26

Ok, I do vaguely remember something about there being different packet sizes possible... That's probably why Dave was having trouble... I'll assume he wasn't using Hyperterminal.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Rayman · 2010-04-23 12:28

Here's another YMODEM code that sends data from HUB ram over to the PC as a file:

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Rayman · 2010-04-23 12:30

PS: I think Dr_Acula is right about not getting much more gain at higher bauds...· But, I do recall the higher bauds being faster nevertheless.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Dave Hein · 2010-04-23 18:06

Rayman,

I am using the Windows Hyperterminal, and it is sending 1024-byte packets.· However, I added some debug code, which was causing me to lose data without a large serial receive buffer.· I was able to get it to work at 4,800 baud with my debug code.· I'll remove my debug stuff and try it at a higher rate tonight.

I did run a loop-time test this morning with the following two loops.

· ser.rxfill("A", 10)
· repeat j from 0 to 9
··· i := ser.rx
··· crc := UpdateCRC(i, crc)
··· pdata[noparse][[/noparse]j] := i

· ser.rxfill("A", 10)
· repeat j from 0 to 9
··· i := ser.rx
··· pdata[noparse][[/noparse]j] := i

I created a modfied version of FullDuplexSerial.spin with an rxfill method that inserts bytes into the rxbuffer.· This was needed to run my timing test.· I was going to attach the test code to this message, but I forgot to bring it to work.

The first loop takes about 400 usec per byte.· The second loop is 5 times faster at about 80 usec per byte.· This corresponds to baudrates of 25,000 and 125,000.· The nearest standard baudrates that could be used with the two loops are 14,400 and 115,200.· So it is better to first buffer up the initial packet and the 128-byte packets, and then perform the CRC computation like you do for the 1024-byte packets.

Also, the ACK for the initial packet should happen after the file is opened instead of before.· Otherwise, received bytes could be lost if it takes a long time to open the file.

Dave

Rayman · 2010-04-23 20:29

Ok, my memory has faded a bit on this... But, I think Hyperterminal gives you a couple YMODEM options... I vaguely remember that one works and one doesn't... Perhaps you're using Ymodem-G with 1k packets instead of the plain "Ymodem" ?

Just a guess. It's been a long time since I looked closely at this...

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
My Prop Info&Apps: ·http://www.rayslogic.com/propeller/propeller.htm

My Prop Products:· http://www.rayslogic.com/Propeller/Products/Products.htm

Dave Hein · 2010-04-24 04:46

I got it to work at 57,600.· I didn't try it at higher rates, but the transfer rate is limited by the write speed to the EEPROM file system.· I did restructure the code so it captures the packet first, and then computes the CRC.· I also added a 10 second timeout at the start of each packet, and a 1 second timeout on each byte within the packet.· This way it doesn't get stuck in an infinite loop if it misses a byte.

cheezus · 2019-02-14 00:29

Bumping this thread because I've been using this tool for a while, and I've made some modifications I wanted to share. They're built on the first release, I believe. I'll look at the differences between the last posted code and my modified code but shouldn't be too hard to reconcile any differences...

So what's so different about my version? The big one is I chose to use Kye's SD instead of fsrw. I did this because Kye's is used in the touchscreen os and I wanted a tool I could copy files to and from the sd card without removing it from the system. I had it working a long time ago but it was buggy and I recently decided to try to fix the issues. There may be a few bugs left but it seems rock solid now. I also added a hook to let you run a program from the sd card (kye's chain function).

The other major change was adding a way to set the baud so I don't have to recompile each time I wanted to test a new baud. This is going to be handy while verifying Bluetooth (and maybe WX) modules. I'm not real happy with the way I wrote it but it seems to work.

PUB NewBaud  | ba, key
  ser.str(string(27,13,10, "[2J  Select new baud: ", 13,10))
  ser.str(string(               "=====================", 13,10))
  ser.str(string(               "1        =       9600", 13,10))
  ser.str(string(               "2        =      19200", 13,10))
  ser.str(string(               "3        =      57600", 13,10))
  ser.str(string(               "4        =     115200", 13,10))
  
  key:=0
  repeat until key== "1" or key== "2" or key== "3" or key== "4" or key== 27
    key:=GetKey

  case key
    "1" : ba := 9600      
    "2" : ba := 19200
    "3" : ba := 57600
    "4" : ba := 115200   
    other:
      AbortMsg
      return 'abort

  ser.str(string(13,10,"Setting new baud  "))
  
  case ba
    9600   :  ser.str(string("= 9600", 13,10))
    19200  :  ser.str(string("= 19200", 13,10))
    57600  :  ser.str(string("= 57600", 13,10))
    115200 :  ser.str(string("= 115200", 13,10))

  ser.str(string(13,10,"Press spacebar to continue or ESC to cancel"))

  key := 0
  repeat until key== 32 or key== 27 
    key:=GetKey
  if key == 27
    AbortMsg
    return 'abort
    
  'waitcnt(cnt+(clkfreq))
  ser.start(RXPIN,TXPIN,0,ba)                   'open comm port


  key:=0
  repeat until key==32
    waitcnt(cnt+clkfreq)
    ser.str(string(27,13,10,"[2J Hit spacebar to continue"))
    repeat 
      key:=ser.rxcheck
    while key>0 and key<>32

There's some optimizations with handling user input, these could use some further work. I'll revisit optimizing and modularizing later. The previous version I used was a separate "app" that needed to be loaded from the desktop but I'm trying to include it in some builds of the os. I think I have that working but once again, it's not pretty. This is where optimizing is going to help because the OS is quite large already and saving every byte counts.

PUB Strap
  Start("y")
  repeat
   
PUB Start(standalone)
  alone := standalone
  if TXPIN
    if alone == "n" or alone == "N"
      cog := cognew(Run,@stack)  +1
    else
      Run
      
PUB Stop
  ser.str(string(27,13,10,"[2J Quitted",7))
  waitcnt(cnt+clkfreq)      
  ser.stop
       
  if alone == "y" or alone == "Y"
    sdfat.unmountPartition
    sdfat.FATEngineStop
  else         
    if cog
      cogstop(cog~ - 1)   
    
PRI Run| i

  num.init                                            'start numbers
  ser.start(RXPIN,TXPIN,0,DEFAULTBAUDRATE)                   'open comm port

  'Establish serial link (wait for client to hit spacebar)
  i:=0
  repeat until i==32
    waitcnt(cnt+clkfreq)
    ser.str(string(27,13,10,"[2J Hit spacebar to continue"))
    repeat 
      i:=ser.rxcheck
    while i>0 and i<>32   
 
  if alone == "y" or alone == "Y"
    sdfat.FATEngineStart(SD_Basepin, SD_Basepin+1, SD_Basepin+2, SD_Basepin+3, -1, -1, -1, -1, -1)
    i :=  \sdfat.mountPartition(0)
    
  repeat
    Debug(i)
    i := \Menu

I think there may have been a bug in the original version, when I was testing my SD card modifications error messages weren't being displayed. The packet was ACK ed before the file was opened, I think I have it correct here. Hopefully that was corrected in the last release. I also have some head scratching with regards to timing. Seems there needs to be a break before sending EOT and between EOT and sending the error message.


    if done== 0    
      if overwrite  
        \sdfat.deleteEntry(@fbuf)
      sdfat.newFile(@fbuf)
      sdfat.openFile(@fbuf,"W")

    ser.tx(ACK)
                     'open output file
    if done==1
      waitcnt(cnt+clkfreq)
      ser.str(string("Batch file reception complete."))
      return 

    ser.tx(Cee)                 'receive packets


PRI Debug(msg)

  if strsize(msg) > 2
    \sdfat.closeFile
    waitcnt(cnt+clkfreq) ' need 3 seconds or message disapears? works with 1 /2 combo
    ser.tx(EOT) 
    waitcnt(cnt+clkfreq*2) ' at least 2 or message disapears 
'    ser.str(string("DEBUG"))
    ser.str(string(13,10))
    ser.str(msg)
    
    Waitkey(32,string(13,10,"Press spacebar to continue",13,10,7))

This needs a good cleanup before I consider it "release quality" but I wanted to share it since the original was so valuable to begin with. Tested working from 9600 to 115200. I didn't try any higher baud rates yet. Any thoughts or suggestions greatly appreciated!

*edit -

I just did some testing to compare the version I posted to the original transferring a 244k file @115200 baud. Note times are similar and could be me hitting start on a phone stopwatch at different times. The final test is the original code compiled with fastspin. Kye's SD card driver won't compile with fastspin

@115_200 baud - extra putty- Send to SD
(250,094 bytes)

Regular SPIN - Kye's SD
129:52 seconds
1,930.9 k/s

reg spin - fsrw
125:29 seconds
1,996.1k k/s

fast spin - fsrw
0:44:94 seconds
5,443.9 k/s

msrobots · 2019-02-14 15:37

@cheezus,

I like @Kye's driver because of it's completeness, but FSRW is about as twice as fast. Some years ago I made a RAISD-Kit, aboard plugging into the quickstart board providing 4 SD-card slots.

At that time I did extensive speed tests and @lonesock's SPI block driver seems unbeatable.

Enjoy!

Mike

cheezus · 2019-02-14 19:18

@msrobots

The most interesting result to me is that using the regular spin compiler (PropTool), @Kye's driver is almost as fast as fsrw. To me that says the the SD driver is probably not the weak link. It's likely the CRC where things are slowing down. I might hook up the LA and explore further a bit later. I tried pushing the baud up to 230,400 and things stop working so it would be interesting to see what's going on the wire. Just guessing that FDS buffer overflows on the first 1024 byte packet. I thought I had a bigger buffer version somewhere.

I think the FastSpin compiled program is probably pushing what this protocol can do, on this chip... I think 5k is about the best that can be expected with Ymodem-1k. There's some other protocols that have better potential but they're more difficult to handle for a number of reasons. There might be some small performance gains by trying to parallize tasks but the protocol is very linear in nature.

I prefer kye's driver because it's completeness as well. I also use the Chain function a lot. Call it a poor man's BigSpin.

I have this idea of trying to build a debugger based on the Ymodem terminal program I'm working on. Something similar to Viewport but with file transfer, mostly for developing with Touch.spin OS. I've got a lot of other things to finish before I begin something like that though. Speaking of viewport, I really wish I could find my key but that seems long gone.

I've focused on the Send function, since I mostly want to send BMPs and EXEs to the sd card. At some point I might work on the Receive function but unless others are interested I have bigger fish to fry. My goal was a tool so send a compiled binary and it's associated files to the SD card and that's pretty much done. Implementing batch transmissions is probably the next thing so I can send more than one file at a time.

Thanks again @Rayman for this awesome tool. Wired or wireless, it's so nice to not be removing the SD card every 5 seconds!

msrobots · 2019-02-14 21:10

@cheezus,

I have to say that I am very impressed by you grasping the propeller so fast. It is a joy to read your posts, I am currently busy with other work and can not allow myself to start a new project right now, Sadly I lost all sources for my RAISD project for stupidity but I had a build a Version of Kye's driver using Lonesocks PASM, that was quite fast and I had added the chain function to FSRW.

For the RAISD system I split both into two parts, one running on the RAISD propeller accessing the 4 SDs, one for the 2,4 or 8 wire connected Clients. So software written for FSRW could use the FSRW client and software using Kye's driver could use Fat_Engine client. Independent of each other.

Sadly some windows update about Meltdown or whatever prevents my computer from booting by rebooting and I can't get to the files.

Enjoy!

Mike

cheezus · 2019-02-14 21:50

@msrobots,

Thanks, your posts have actually sparked some of the ideas I've had. I've actually been programing the propeller for a while. I did take a "break" and I'm just trying to get my "chops up" again. I used to post under the SN Average_Joe but lost access to the associated email account, so lost viewport, my steam account.. It's surprising how many things can be lost without the associated email account. I also lost my entire library of code so I'm trying to rebuild everything and bug test what I do have working.

I'm trying to get some of the essential functions together to prepare for porting to P2. I really want this Ymodem for P2 but will need FDS and FAT driver. Both are way above my head right now. I somewhat understand the P1, but the P2 is still a mystery to me. I've done some playing around but I'm a bit scared of blowing up my p2 ES. I think I've got 8 or 10 dead P1dip40s from doing stupid stuff over the years. Can't do that with the P2.

I'll keep in mind re Kye's driver using Lonesocks PASM. Moving forward I need to see what exists, if anything for the P2. For now the plan was to start inserting conditional compiles to allow changing of sd driver with 1 source line code. I'm going to be using conditionals a lot so this will be a good place to start that testing.

What sparked this was working on the SVF player, I needed files to test and my current computer has no SD card reader. I need to use an mp3 player to get access to the cards (and I'm limited to my micro cards right now too). I can handle slow transfers, I think it's more about ease of use vs speed. If I start sending multi-megabyte files then I might need to rethink thinks. Speaking of which, I need to get back to the SVF player.

msrobots · 2019-02-14 22:28

I have a working 2-port serial driver running in a COG ad having the buffers in the COG for the P2. It is working but needs some cleanup to get HUB/LUT transfer faster.

For the P1 there is a JDserial (OBEX) driver doing that for one port and I do love that thing. once started it just needs its pins and two longs for the mailbox in the HUB.

The nice thing is that you can use this driver from Spin AND Pasm easy.

My goal is to have something similar on the P2. http://forums.parallax.com/discussion/comment/1464110/#Comment_1464110

Has the latest sources

I split the driver into two files, one containing the PASM one containing the access routines in Spin.

The other Files are the current test suite.

I am busy with other stuff but hopefully can spend some more time on it this weekend. But the current driver works fine with 2 pairs of RX and TX I just want to optimize the HUB/LUT transfers from byte to long when possible on block transfers.

I really could use this Ymodem on the P2, mine has the enclosure and it is quite annoing to reach the SD card.

EDIT and yes I also lost my viewport key for the same reason

Enjoy!

Mike

Wuerfel_21 · 2019-02-17 12:04

Thanks for bumping this thread! If I can get this to work, hours of SD card swapping may be saved...

YModem (2019 update)

Comments