PropForth 6 development

prof_braino

Propforth 6.0 development

Note PF5.5 is still the available public release. PF6 will not be available until mid 2014 2016.

(update November 1 2015) Propforth6 repository is on Github.

https://github.com/PropForth6/PropForth6

Notice that the MASTER branch is empty. All work is happening on the DEV branch; when completed this will become MASTER.

If you are not on the team, you probably don't want to mess with it yet. If you DO want to mess with it, consider joining the team.

Propforth 6 is pretty much the same as PF5.5. Repository is changed to github and the build automation add flow control on both ends. This allows FPGA versions of the prop to be tested with same automation.

Propforth 6.0 adds multiprocessing and load balancing. Sal had an application that needed more than 8 tasks. The number of tasks that need to be active at any time varies from 20 to 30 tasks. Since the Prop 2 is not availible, the FPGA prop 1 was used.

prof_braino

Forth can support software multitasking, even on single core single thread processors. Tasks are defined as having unique stack and user variable space, usually allocatted from physical memory. The tasks are set in a round-robin. Each task executes whatever it needs and takes as long as it needs in one pass.

When a task gets to a slow operation where it has to wait anyway, it gives up control to the next task. As long as each task does not hog all the processor time, this method works really well. It was discovered that loops are slow, and can take a LONG time, so at the end of each loop iteration, so tasks also surrender control at the end of each loop. In this fashion, a forth system can give the impression of multitasking, even if the hardware does not support multitasking.

[in the old days, using forth inc polyforth, we used to get 30 terminals running on an IBM PC 386, and each terminal looked like it had full control of the system. Only when several users tried to access different areas of the flopy disk at the same time was there any indication that the system was being shared. Neat!]

prof_braino

Sal used this same round-robin idea as the starting point fo PF6 multitasking. All 8 cogs are considered as a "pool" of execution opportunities. Cog 7 is used for the serial driver as usual, so we have cogs 0-6 remaining in the pool. The system consists of the pool of cogs and the round robin of "tasks". The next free cog takes the next task in the queue. Each task executes one pass, and the task goes to the end of the queue, and the cog goes back into the pool.

There are a couple cool differences due to the prop hardware. One is that we don't have to give up control. If a cog gets occupied by a very long task, that is fine. It simply is not available in the pool for that amount of time, and the capacity of the prop 1 system is reduced by 1/7th. On the prop 2, capacity would be reduced by 1/32.

Using the prop 1, the demo has 128 tasks. This is the most tthat seems to fit and still alow something to be done. In this demo, the first task recieves a character, and sends it to the next task. The task round-robin and cog pool take just under 700 ms for one iteration of all 128 tasks.

Sal's application had up to 30 tasks, had plenty of capacity to do complex motion control and interestting numberical calculations. He's been running his tests since April (2013), and it looks pretty stable.

prof_braino

Propforth 6 has two sets of kernels: the single task kernels as before, and the multitasking kernels.

The regular versions are optimized and use much less space. (129 cog longs in devkerel)

In the multitasking versions, the assembler is not relocatable.

The kernels are the absolute minimum. the multtasking support take a lot of space, so use we will be doing a lot of SCRIPTS from SD or EEprom.

The error handing is different:

Prop0 Cog6 ok
.

RESET: 6 3 ?    0 ?
Prop0 Cog6 ok

the first number is the cog. The second number is error code (assembler) or the execution address of error (forth), so we can figure out the actual word that failed. helps in debug background execution of the multitasking.

the last number is error data. usually 0 (this might change)

Error codes are

1 stack overflow
2 return stack overflow
3 stack underflow
4 return stack underflow

PFA of failing word....

Prop0 Cog6 ok
: test ERR ;
Prop0 Cog6 ok
test

RESET: 6 16442 test    0 ?
Prop0 Cog6 ok

stacks are assembler failures, other are forth failure

instead of "words", the routine to display dictionaryword list is

wl

prof_braino

Windows comm ports:

If you have too many serial devices you have troubles. If you get weird behaviour, check the comports.

I was up to com37, these were either USB virtual serial ports from the FTDI chips, or from the cheap bluetooth dongles. Anyway there were too many.

Open up device manager, show hidden devices, and unistall all the extra com ports.

When you re-insert a USB device, it will re-assign a comport.

prof_braino

Notice that Propforth 6 implements flow control. With flow control ON, we don't have to worry about 'fl' (the fast load or file lod word). Both sides know how to wait for the other.

In goshell.bat, there is the line

FLOWCONTROL=1

This turns on flow control.

If the display appears to hang when using tera term directly to the com port, hit ACK (ctl^A). The ACK tells the prop to send more characters. Run the words wl or words and you will notice that is is about one buffer worth of character and then you need an ACK.

The flow control works with goterm and goproxyterm; both sides will wait. No more overruns, regardless of the speed of the PC!

prof_braino

I found that the build automation fails on certain PCs. Has anybody else seen this? Message me, please.

On both the Windows 7 netbook and the XP virtualbox session under mint petra, buildall.bat stops after spinmaker. All the other machines run fine, start to finish.

On the netbook, buildkernel.bat appeared to run, but windows decided to install updates and reboot midway through the test, so the only thing I know is I still hate windows.

I'll continue testing both of these rigs for a bit and see what I come up with by Sunday.

frida

1 run fault in StartKernel.spin, look at Udklip01.PNG and Udklip02.PNG.
2 run fault in StartKernel.spin, look at Udklip03.
3 run fault in mpMinEEpromKernel.spin, look at Udklip04.
4 run fault in StartKernel.spin, look at Udklip05.

I am now on linux, and it seems to have a better success.

I will make it run on linux first, if I can, and then try in Windows again.

I would say, it was easier with the first version.


EDIT:
It is now running on my first linux. I will continue with the other linux later.
Now it is walk time with the dog, and then bedtime.

prof_braino

frida wrote: »

.... run fault...

I am now on linux, and it seems to have a better success.
I would say, it was easier with the first version.

The "excedes by x longs" messages usually happen when modifying the kernel. The build automation calculates the size based on the max memory size and the use selections for stack, etc. These messages do not appear using windows (with unmodified kernel), did you change anything, or can you give us steps to reproduce the issues?

We have found items that need investigation in the protypes that were sent out. The build automation fails on netbook and on linux virtualbox windows XP sessions. Goterm works fine but goproxyterm has trouble on the netbook. Errors might be in the set up process, or may exist in the GO code.

We may have a new prototype by this weekend.

Thanks for the responses so far! Please continue to play.

frida

The only thing I done, was delete all in bin and all in pkg.
Then I run go install goterm, and then I run the installation.
Nothing else was modified.

frida

Good news.
The installation is now running on LUBUNTU 13.10, UBUNTU 13.10 and WINDOWS 8.
The fault was one of my Propeller Plug, I have 2, bought at the same time.
1. FTDI 0814-B, FT232RQ, was the one that failed.
2. FTDI 0643-A, FT232RQ, is the one that works.

The faulty, only went to a line between 3000 and 6000, where it then missed some characters, then it continue and make the file complete.

frida

Well, I'm almost through on windows 7. It is running op to buildall-23 now, with small changes in some of the files.

From:
t 3000 CO
\
\ Done
\
CR CL

to:
t 3000 CO CR CL
\
\ Done
\

But the main change was in windows_serial.go. I changed the line:
from: dataout := make( []byte, 256)
to: dataout := make( []byte, 32).

Why is it 256? It can max recieve 128 bytes.
From my datasheet on ft232:

FIFO RX Buffer (128 bytes). Data sent from the USB host controller to the UART via the USB data OUT
endpoint is stored in the FIFO RX (receive) buffer. Data is removed from the buffer to the UART transmit
register under control of the UART FIFO controller. (Rx relative to the USB interface).
FIFO TX Buffer (256 bytes). Data from the UART receive register is stored in the TX buffer. The USB
host controller removes data from the FIFO TX Buffer by sending a USB request for data from the device
data IN endpoint. (Tx relative to the USB interface).

prof_braino

frida wrote: »

But the main change was in windows_serial.go. I changed the line:
from: dataout := make( []byte, 256)
to: dataout := make( []byte, 32).

Why is it 256? It can max recieve 128 bytes.

Sal might have missed something here, he's checking into it. The way the queue is handled was supposed to take care of this. Sal testing indicated that the is some timing issue someplace, and the "second" transmission can loose the first character. The result is the USB port locks, we have to pull the cable and reinsert to clear it. Sal's failures are intermittent, mine are pretty much everytime (on the boxes that experience failures) so it might be something else with the hardware.

I tried changing the data out to 32 as above, but it still fails on the systems in question (linux virtual box and netbook). There might be anotther part to this, or more than one issue.

Sal will have another drop for us Monday or Tuesday. I will send it to the team when I get it, but I probably won't be able to test it first. Please post here or PM when you find issues.

frida

Well, I am through 2 passes now on my windows 7, without any zombie or USB port locks. I'll try it on windows 8 also. If it runs well, I will send the changes I have made.

frida

Well, I managed to run 5 passes without a single stop. And run 3 runs in Windows 8 without problems. And now it works both my usb plugs.
I just used the one that failed in windows 7 without problems. Through the races take approx. 56 minutes, on both machines.
I am attaching the files that I have made changes in. I have not had time to clean up the files, but I want you to try if they work for you.
I'm going evening walk with the dog, and bedtime.

frida

Why do some version strings have 1 to 3 "~ h20"
1 without
X (version) dup C@ swap c" EEpromROKernel" (version) ccopy C!
1 with
X (version) dup C@ swap c" ~h20~h20~h20SDKernel" (version) ccopy C!

The line in buildbase.bat has a "/IMAGE" too much. And it dont have "\" like the others.
"del /Q /IMAGE MAKE/results/outputFiles/mpTMPKernel.eeprom"

frida

I now have removed all sleeptimers again.
The dataout buffer are now 8192, and it is working like a charm, even I am working on the PC while it is running.
I have mark all lines with "//**********PBP" where I have made changes, I hope!
So I guesses on there are some hiding i windows on the way to ftdi. gobuffer->windows->ftdi. 8192->xxx->128.
Try it, and incorporate it in the next test. In the meantime I will do som more test on LINUX.

prof_braino

A new build has been sent to the team. If you want to be added to or removed from the list, please PM me.

It turns out that flow control has uncovered an interesting situation. On a very slow PC, the buffers can fill up. when the transmission burst happens, it can choke the hardware send or recieve buffer. Being synchronous, we wouldn't expect this, but on a fast machine, the buffers wouldn't get as full. Sal adjusted the queue length, and change the timing. This eliminates the lockups due to the "first character of the next transmission" issue. The modifications were in the GO code (on the PC side); the propforth kernel remains stable and unchanges since last April. The change makes the slow machines work. (I don't understand completely, I'm just happy the tests run).

The release plan is per usual: Sal will document the internals (in Propforth.htm or equivalent). I will create step-by-step set up instructions based on Sal's directions.
The team may contribute with test results, review comments, questions, any demos or experiments that we can put ut as a web page, or post as code.

What sensor monitoring, preprossessing, local actuator control, and logging application would you suggest, if we have units of 30 tasks, and SD gigs worth of storage?

We're looking for something cool yet inexpensive, that regular folk might actually try.

frida

The new version failed on my windows 7.
It still makes zombie and USB port locks
I will stick to my version, until a new one gets released.
See the pictures.

prof_braino

frida wrote: »

The new version failed on my windows 7.

Could you please post your hardware for the windows 7 system? CPU + motherboard, are the usb built into the motherboard? Any info would help. Thanks!

Also, on mine, I had failure the first time I ran it. I think it was operator error (but you might be showing otherwise). Is you error repeatable, can you get it to happen on another system? Thanks!

prof_braino

You second picture is exactly where my systems would stop with a locked USB port, and it would not restart until the quickstart was unpugged and replugged.

Can you find the change in the delay in the go code? I'm just guessing, but that might be a place to start.

frida

As in post:
http://forums.parallax.com/showthread.php/146693-Propforth-v5.5-is-available-for-download/page19#380

I sent my suggestion in # 17, which solved my problem.

prof_braino

frida wrote: »

As in post:
http://forums.parallax.com/showthread.php/146693-Propforth-v5.5-is-available-for-download/page19#380

I sent my suggestion in # 17, which solved my problem.

Thanks, entered issue 214
http://code.google.com/p/propforth/issues/detail?id=214