Audio Forth on the Propeller -- something like the Ironman's computer interface.
LoopyByteloose
Posts: 12,537
in Propeller 1
This is all in the conceptual stage. Generally I am a purist and desire to build Propeller projects that aren't dependent on other devices. This is a hybrid project.
The concept.
Be able to talk into a microphone, have your speech collated into good Forth and passed on the Forth on the Propeller via RS232 serial interface. A wireless link might be created either Wifi or Bluetooth.
A. Forth on the Propeller is mature enough that there is almost nothing difficult about the creation of a dictionary additions that might be required.
B. Speech-to-Text in Linux, Windows, or Android would have to be adapted for passing on Forth commands.
>>> At this point I am mostly considering using an Android device speech-to-text resources and would prefer to use wifi as a wireless link.
I have both a wifi and bluetooth line that converts to serial that is hard-wired to a Propeller. But the wifi link handles more distance better and may be easier to send text as a .txt file to be opened and sent word-by-word to the Propeller at the receiving end.
C. Have the Propeller reply via text-to-speech
>>> It would also be wonderful to have the Android receive text and read it too me as the Forth on the Propeller would then seem to speak to me.
+++++++++++
Something all the same lines could be done with another OS and wireless is optional. But the real goal here is to have the beginnings of a user interface that is extensible by speech converted to Forth words.
IOW, program and operate your Propeller entirely by talking and listening.
+++++++
For instance, one could say Colon, then a name, followed by a Forth word or value, and so on and close with Semi-colon. But it might be much clear to create your own more natural dialogue defined that starts with LISTEN (generates a Colon and waits for other items) and closes on ENOUGH (generates a Semi-colon).
It certainly would take some practise getting used to and developing habits for confirming what you are doing. But it is very feasible. Seems like a lot of fun to me.
The concept.
Be able to talk into a microphone, have your speech collated into good Forth and passed on the Forth on the Propeller via RS232 serial interface. A wireless link might be created either Wifi or Bluetooth.
A. Forth on the Propeller is mature enough that there is almost nothing difficult about the creation of a dictionary additions that might be required.
B. Speech-to-Text in Linux, Windows, or Android would have to be adapted for passing on Forth commands.
>>> At this point I am mostly considering using an Android device speech-to-text resources and would prefer to use wifi as a wireless link.
I have both a wifi and bluetooth line that converts to serial that is hard-wired to a Propeller. But the wifi link handles more distance better and may be easier to send text as a .txt file to be opened and sent word-by-word to the Propeller at the receiving end.
C. Have the Propeller reply via text-to-speech
>>> It would also be wonderful to have the Android receive text and read it too me as the Forth on the Propeller would then seem to speak to me.
+++++++++++
Something all the same lines could be done with another OS and wireless is optional. But the real goal here is to have the beginnings of a user interface that is extensible by speech converted to Forth words.
IOW, program and operate your Propeller entirely by talking and listening.
+++++++
For instance, one could say Colon, then a name, followed by a Forth word or value, and so on and close with Semi-colon. But it might be much clear to create your own more natural dialogue defined that starts with LISTEN (generates a Colon and waits for other items) and closes on ENOUGH (generates a Semi-colon).
It certainly would take some practise getting used to and developing habits for confirming what you are doing. But it is very feasible. Seems like a lot of fun to me.
Comments
: redefined as Hmmmm
; redefined as yes?
Hmmmm add3 3 + . yes?
*Google's speech to text is rather impressive. I'm not sure if Android has a native speech to text but you could always ship it off to Google and get the text back and then pass it to your propeller...no?
How would you have the Android device generate 'add3' from speech to text? You might have to go to some sort of 'spelled mode' as opposed to saying 'addthree' which might get parsed into 'add three'.
In other words, there are going to be snags over the simplest aspects of what you take for granted. The whole purpose of the project is to find out if anybody can comfortably and usefully program by talking alone.
+++++++++++++++++
Android has native Text to speech and Speech to text available, and I have an Asus ZenFone that seems to support it.
It that becomes too much, I'll try a Linux text-to-speech and Linux speech-to-text pair. Though Google offers a similar set that may work for Apple, MS, or Linux.. not yet sure.
I had lunch with a friend yesterday that claimed he uses the Text to Speech function to create chapters for a book he is writing. Should I believe him?
Anyway, the ZenFone has both Wifi and Bluetooth, but the Bluetooth is not quite what I had hoped for. Range is limited. Also, I haven't found the serial terminal application that I would be satisfied with. I am hoping Wifi in Android will allow me to do better.
In short, the hardware scheme is this..
ZenFone <---wifi-->hacked TP-Link MR-3020 Wifi modem <--USB serial-->Propeller 1 Projectboard.
The ZenFone has what I think is Android 4.4
The TP-Link MR-3020 is reloaded with OpenWRT Linux and Heater's Propeller utilities.
The Propeller can use any of the 3 Forths - Tachyon, PropellerForth, or pfth. (I think I will use Tachyon.)
At this point, my main concerns are with getting a verbal way to trigger the sending of text files within the Android. It would defeat the whole scheme if I have to work back and forth between touch screen and speech.
And get the MR-3020 to be automate transfer processes. It has been awhile since I used my MR-3020 and as best I recall I could not have wifi working unless I also had a LAN plugged in. I need to resolve that configuration bug so that the Propeller could be mounting on anything mobil and autonomous.
It seems it can work. And for just regular spoken speech it works remarkable well - both when I speak English and Chinese. I certainly could write email or a book by talking to it. The learning curve is mild.
But Forth is not English, nor Chinese and that upsets it a bit. It doesn't appreciate punctuation being in unpredicatable places or the un-grammatical; so at the very least, the Colon and Semi-colon in Forth are better off adapted to predictable words spelled out in English. Numbers might be written out as words as well, so Forth will have to convert in those situations.
Once I get going single Forth word commands should not be too difficult.
Entirely spoken entry is not yet feasible for me. I generate text, then have to do something to send it. And I may have to edit by hand before sending.
Previously, I found that I did NOT like using my ZenFone with a terminal program and the tiny keyboard as a Forth wireless terminal.... too many typos with the poke poke entry. That was a huge disappointment. But this makes the task much more pleasant. Nonetheless, it is NOT as seamless as what you saw in Ironman films. There is a bit of back and forth.
Conclusion -- There are enough good things happening that I will continue to try and find a way to work with the ZenFone. At the very least, it will allow me to discover more about how I want to use wireless Forth and an Android device.
ShortTalk: Dictation Made Rewarding
1) Find someone whose English is not so good. I believe you have many of them around you.
2) Be sure they don't know Forth, or whatever programming language you are trying to dictate. For sure you have many of those around you.
3) Dictate a program to them and have them write it down.
Check the resulting text.
This seems like a problem that is way off the radar in complexity.
It can be simplified by insisting on phonetic input of individual characters "a", "b", "c"...but that sounds very tedious and more effort than it is worth.
Unless...you are making this for blind people. I have met a couple of blind programmers, it's amazing how they get along.
My Android 4.4 does surprising well with my English, and my Chinese. I was quite surprised that it did so well with my Chinese speech.
Still, it has problems... like one, two, three becoming "one two tree" or "123" without the spaces that I desire.
So the biggest snag so far is that numbers are somewhat randomly spelled or converted to numerical notation. I presume there is some sort of context algorhytm that dislikes the Forth interferrring with more native speech context.
++++++++++
I have been working BOTH with a Bluetooth link and a Wifi link as my Bluetooth Forth machine is up and running 24/7. But so far, I haven't located an Android Bluetooth terminal that will cleanly handle input. Again it is a problem with missing spaces. I am using a Propeller Protoboard with an HC-05 on a 12V gel cell with 24 hour trickle charge.
My wifi set up adds more effort on the reception end to create a conduit to relay the serial text through the MR-3020 to the Propeller. Right now I am frustrated as I apparently installs SSH serial and what forgotten the Admin password. So it seems I have to reload the MR-3020 OpenWRT and work through reconfiguration to get what I need. It is a distraction, but needs to be resolved.
===========
Will Forth on the Propeller ever take natural speech in a fluent manner from Android. I doubt it. But it can handle listening for key words. There may be a way to filter other conversation in a loop. But at this point, it doesn't seem to care at all about voice recognition. So use seems to be viable only when one speaker is present.
For now, it seems that I need to rebuild my MR-3020 configuration to move forward.
Thanks... I will consider anything at this point. While I am working mostly with my ZenFone Android 4.4, maybe I can make more headway with ShortTalk in Linux. I have a newbook computer with Debian that has a microphone, the wifi, etc.
Google Speech-to-Text just presumes that the end user only desires to write a document, not have more precision to control something.
++++++++++++
But the "holy grail" seems to be a software application that will NOT only to text-to-speech. It must also recognize the speaker with voice recognition and exclude all others, or many each addition in a different dialog.
While obvious, nobody seems to be attempting a combined voice recognition and speech-to-text server.
Forth may not be the ideal programing language, but it certainly does get the ball rolling and is revealing areas that need development.
A special command added, for example "forthword".
When the the speech to text app detects the special control word "forthword",
it looks for ONLY words that exist in the forth dictionary.
The exception would be of course the word colon (for ":" used for starting a new forth word definition).
When colon is heard, it would have to do what ever to determin is you mean "D123" or "D 123" etc. This would be something like the forth word ACCEPT which takes whatever follows from the input stream, but this is getting into inplementation detail, and it early for that.
Anyway, when the app hears semi-colon, it can terminal a forth colon definition as normal. Then it goes back to looking for words that match forth dictionary entries so you can execute you defined words.
I would guess "end-forthwords" would work to switch back to non-forth text to speach.
The big issue would be the feedback, for example after the work "colon" above. How to voice edit the input if the first guess is incorrect? A voice based text editor sounds like a blast. I would need a college kid that does android programming to work on this, I would not have the bandwidth to do it myself. But it sure sounds fun.
Something like the following...
: listen ." : " space ;
: finish ." ; " ;
I still have trouble with numbers. I say 'three' and get 'tree'. I say one two three, and I get '123' which the Forth will see as 'one hundred twenty three'.
So there are quite a few issues to be worked around.