Shop OBEX P1 Docs P2 Docs Learn Events
Tool to convert Tokenized Sping 2 text? — Parallax Forums

Tool to convert Tokenized Sping 2 text?

davidsaundersdavidsaunders Posts: 1,559
in Propeller 1
In order to get back into the swing of things I was going to look at some of the code in OBEX. I quickly found a problem, namely that most of the code is tokenized, and I do not have any way of viewing it.

As such is there a tool that will run on ARM Linux that can convert the tokenized code into a text listing? Without having to install an IDE or editor?

Comments

  • ElectrodudeElectrodude Posts: 1,657
    It's not tokenized. It's encoded as UTF-16LE for some reason. Your ARM Linux should come with iconv or something similar that can fix it for you. Also, most text editors should be smart enough to automatically read it properly.
  • Heater.Heater. Posts: 21,230
    I'm very sure that is not true. All code in OBEX is source code. Human readable and released under the MIT licence.

    However, Spin source code is in some variant of Unicode. I suspect that is what is tripping you up.

    I don't recall the details but I suspect it is some kind of 16 bit character encoding.

    This drove me nuts when trying to run OpenSpin in the browser.
  • davidsaundersdavidsaunders Posts: 1,559
    My text editors do deal with UTF-16, the problem is that at least 4 of the examples I downloaded are tokenized, I once before ran into this problem, and it was solved by biting the bullet and using the Propeller Tool, I was just hoping that now that the Propeller Tool is not the biggest show in town someone had already made a de-tokinizer for the older tokenized code.
  • davidsaundersdavidsaunders Posts: 1,559
    Heater. wrote: »
    I'm very sure that is not true. All code in OBEX is source code. Human readable and released under the MIT licence.

    However, Spin source code is in some variant of Unicode. I suspect that is what is tripping you up.

    I don't recall the details but I suspect it is some kind of 16 bit character encoding.

    This drove me nuts when trying to run OpenSpin in the browser.
    I do not remember the specifics, though I do remember that there was a thread way back in 2010, or there about, that dealt with the older tokenization, that everyone was trying to avoid at the time.

    I do not know what to do about it now.

    Though I will give a try with other UTF capable tools, just in case it has been converted to some form of UTF.
  • davidsaundersdavidsaunders Posts: 1,559
    Ok right after making the last post I tried one more editor, and it opened it without any trouble (other than missing characters from the extended Propeller Font).

    So it is some kind of UTF encoding, weird that other UTF-16 capable editors did not work.
  • Heater.Heater. Posts: 21,230
    David,

    Which 4 of the examples are you talking about?

    The Prop Tool and hence OpenSpin, Simple IDE, Propeller IDE, BST, HomeSpun have no idea about linking compiled/tokenized objects into a program.

    So there should not be any such things in OBEX.

    There has been talk of OBEX having been hacked and spammed recently. Perhaps you are suffering from that.

  • Heater.Heater. Posts: 21,230
    Ah, we are cross posting.

    You remind us of a good point. Spin source uses a weird encoding that should be fixed for the new Spin 2 for the Propeller 2.
  • Cluso99Cluso99 Posts: 18,069
    Sometimes source is saved as 16bit encoding by PropTool, particularly if you use the propeller font for doing graphics. This is often used for showing diagrams within the source code.

    IIRC there was a way to convert back to 8bit encoding but of course the graphics (which are only comments) will be somewhat lost.
  • Heater.Heater. Posts: 21,230
    Well there is the thing.

    Whatever graphics symbols are required no doubt exist in Unicode.

    So if Spin source was saved in UTF-8 there should not be any loss.

    If those symbols do not exist in Unicode then the should be disallowed in Spin 2 and replaced with whatever is available.

    Of course the whole idea of doing such graphics in source text is broken. It relies on the the reader having the right font to display it correctly.
  • jmgjmg Posts: 15,173
    Heater. wrote: »
    You remind us of a good point. Spin source uses a weird encoding that should be fixed for the new Spin 2 for the Propeller 2.
    Cluso99 wrote: »
    Sometimes source is saved as 16bit encoding by PropTool, particularly if you use the propeller font for doing graphics. This is often used for showing diagrams within the source code.

    IIRC there was a way to convert back to 8bit encoding but of course the graphics (which are only comments) will be somewhat lost.
    Heater. wrote: »
    ...
    Of course the whole idea of doing such graphics in source text is broken. It relies on the the reader having the right font to display it correctly.

    Maybe this needs a thread in P2 forum, about Source rules and Fonts ?
    Certainly, requiring a font is just nuts these days, and for any Graphics in source files, I use this tool :

    https://josoansi.de/download.php (Looks like they now include source)
    AACircuit allows quite useful schematics in standard font.
    To render on almost anything, use any fixed-space font.
  • davidsaundersdavidsaunders Posts: 1,559
    I rather like the Propeller font, it saves space when showing schematics on a VGA or NTSC monitor hooked to a propeller.
  • jmgjmg Posts: 15,173
    davidsaunders wrote: »
    I rather like the Propeller font, it saves space when showing schematics on a VGA or NTSC monitor hooked to a propeller.
    Well, yes, it is not going away,
    If you really do develop using NTSC monitor, you are free to use the Prop font & I see it will make sense in embedded systems.
    Just best avoided in web-published source code.

  • Cluso99Cluso99 Posts: 18,069
    I have found the propeller font to be a very useful tool, both as a writer and a user.
    It's just a pity that there seems to be problems with other software/systems. PropTool is fine with it.
  • Phil Pilgrim (PhiPi)Phil Pilgrim (PhiPi) Posts: 23,514
    I like the Propeller font, but outside of the Propeller Tool, I prefer Proggy Clean for programming:

    http://www.proggyfonts.net/download/

    -Phil
  • jmgjmg Posts: 15,173
    Phil Pilgrim (PhiPi) wrote: »
    I like the Propeller font, but outside of the Propeller Tool, I prefer Proggy Clean for programming:

    http://www.proggyfonts.net/download/

    Interesting to see they have a bold punctuation option.
    Anyone tried that ?

Sign In or Register to comment.