URGENT..plz what is the tokenization process means????????

Archiver

plz i want to know what are the tokenz and the process of
tokenization?????



Thx foe all

Archiver

This may have already been answered, but:
'Tokenization' is the process of converting
the ascii source code (human readable) into
something the BS2 can work with.

Each PBasic keyword (like 'SERIN') is converted
by the IDE/compiler to a token (a small integer,
(like 20)) (Note I don't really know what token
they use for SERIN).

The BS2 has code inside it to 'interpret' this
token, and execute the appropriate instructions
to implement a UART RS232 recieve routine.

Every PBasic keyword has an appropriate token
to convert to, which the 'interpret' code
knows what to do with.

This is similar to the process the Java language
uses -- but of course the Java interpreter
code is much larger and supports objects.
My point is compiling to tokens and then
'running' the tokens on an 'interpreter' is a
long accepted way of implementing a language.



--- In basicstamps@yahoogroups.com, "rme90" <rme90@y...> wrote:
> plz i want to know what are the tokenz and the process of
> tokenization?????
>
>
>
> Thx foe all