Oh, I've found the problem with fwrite() too. I never knew about the distinction between items and bytes. I guess I'd always fluked it in the past. That and Flexspin treats them as the same so I'd accidentally got it wrong there and not known it.
Buffer = 2 kB, ................................................................ Written 128 kBytes, duration 13508 ms, makes 9 kB/s
Verified, ................................................................ Read 128 kBytes, duration 842 ms, makes 152 kB/s
Buffer = 8 kB, ................ Written 128 kBytes, duration 13502 ms, makes 9 kB/s
Verified, ................ Read 128 kBytes, duration 840 ms, makes 152 kB/s
Also, I'm seeing an odd behaviour with text sent to the terminal. The writing portion oddly buffers the whole line before emitting it. While the verifying and read speed portions both emit in real time.
Also, I'm seeing an odd behaviour with text sent to the terminal. The writing portion oddly buffers the whole line before emitting it. While the verifying and read speed portions both emit in real time.
Just a question of how much stuff is in the buffer at the time, I think. Add an fflush after each putchar to force it out - i.e. putchar('.'); fflush(stdout);
Yeah, that works. But it's not needed for the same action during fread(). Which I see as strange.
BTW: The second time through using the bigger file buffer is same behaviour - without fflush() the terminal doesn't emit until end of line for fwrite() portion, while for both fread() portions it emits immediately.
Here's an updated version with more dots during the verifying portion.
@evanh said:
I'm thinking fopen() in write mode must be changing something internal to char handling.
What's happening is that fread() automatically flushes all open line-buffered output streams - including stdout - but fwrite() does not.
This appears to be in accordance with the C specification, which says (in part) ...
Furthermore, characters are intended to be transmitted as a block to the host
environment when a buffer is filled, when input is requested on an unbuffered stream, or
when input is requested on a line buffered stream that requires the transmission of
characters from the host environment. Support for these characteristics is
implementation-defined ...
EDIT: Oh, wow, it's only fread() that auto-flushes. Any other time I have to manually do it, irrespective of fopen() mode or anything else.
That's weird, why would fread() even bother? What's weirder is shouldn't writes be the ones to flush more?
@evanh said:
Good point. It would be insane to not use a kit.
EDIT: Oh, wow, it's only fread() that auto-flushes. Any other time I have to manually do it, irrespective of fopen() mode or anything else.
That's weird, why would fread() even bother? What's weirder is shouldn't writes be the ones to flush more?
The C standard says it is so that an fread() after an fwrite() that is intended to display a prompt on a terminal will write the prompt before it does read.
@evanh said:
I don't get why that needs an exception. Why not the other way around also? I'm suspicious it's probably an ancient backwards compatibility thing.
The other way around would mean that output streams could never be line-buffered
Doing it this way around means writes can be line-buffered except when you do a read, in which case the output is flushed even if it is not a complete line (which is typical in the case of a terminal prompt).
The only thing that is a little unexpected is that doing an fread() on an input stream flushes all line-buffered output streams - and the reason for that is that stdio does not know which stream is being used to output the prompt - e.g. when you read input from stdin you actually need to flush stdout to ensure the prompt will be displayed on the terminal before you do the input.
The behaviour of your program is a consequence of that.
@RossH said:
The only thing that is a little unexpected is that doing an fread() on an input stream flushes all line-buffered output streams - and the reason for that is that stdio does not know which stream is being used to output the prompt - e.g. when you read input from stdin you actually need to flush stdout to ensure the prompt will be displayed on the terminal before you do the input.
Is that a requirement by the spec though, or is it just implementation defined too?
Separately, I would like to be able to switch off the line buffering of stdout. Or, even better, have libc built without line buffering at all.
Interrupts aren't used for transmitting, right? That would be an excuse to have a buffer at least.
I finally found a nasty bug that has been annoying me for some time. It popped its head up nearly every time I was just about to do a release, and every time it did so I found a workaround for it - but I could not find the underlying cause.
It turned out to be in my implementation of getenv(), which is a standard C function that reads environment variables. It only affected Propeller 2 C programs which used environment variables, or Lua programs (because Lua uses environment variables during initialization), and it also mostly affected programs that used XMM, because it could cause cache corruption during program start up - which made it difficult to find, because if the program started at all it generally worked fine.
I have already uploaded the fix to GitHub - attached is a patch for non-GitHub users. The fix will also be included in the next release.
This is a full release. It adds eLua support for using the Parallax WiFi module to make Remote Procedure Calls (RPCs) between propellers. Note that demo and test programs may compile for both the Propeller 1 and 2, but they have only currently been tested on a Propeller 2. Also, some of the WiFi functionality requires a firmware update to the Parallax WiFi module (the updated firmware is included in the release).
Here is the relevant extract from the README.TXT
RELEASE 8.5
New Functionality
-----------------
1. Catalina now supports WiFi-based Remote Procedure Calls (RPCs) in addition
to ALOHA serial RPCs. These are now implemented alongside the ALOHA
protocol in several eLua variants. It is possible to have just ALOHA
services, just WiFi services, or a combination of both. All the existing
ALOHA example programs have been updated to use either ALOHA or WiFi,
and an example of using both in the same program is given in a new
'hybrid' example. See the document 'ALOHA from Lua' for more details.
2. The binaries generated by compiling the ALOHA versions of eLua have been
renamed alua, aluax etc to be consistent with the new WiFi RPC versions
of eLua (which are called rlua, rluax etc). No changes to functionality.
3. The eLua programs now load a generic 'serial' module if Lua is compiled
with either the 2 port or 8 port serial plugin library (i.e. -lserial2 or
-lserial8). Lua programs can use the 'serial' module without needing to
know if it is using the 2 port serial or 8 port serial plugin.
4. Lua now includes a module ('wifi') that allows access to the C WiFi support
functions. The module is loaded automatically if Lua is linked with the
'wifi' library (e.g. compiled with -lwifi).
5. All the pre-compiled versions of the eLua demos have been removed, since
they will all need to be modified and recompiled to use the new WiFi RPC
capabilities.
6. Base64 encode and decode routines have been added to the Catalina library.
See include/base64.h for details.
7. The buffer size of the 8 port serial plugin has been increased from 32
bytes to 1024 bytes to accommodate the new WiFi RPC functionality. Since
there are 16 such buffers (one for each direction of each port) this can
make programs that use this plugin up to 16k larger. If this causes
problems, the files target\p2\s8serial.t and source\lib\serial8\core.c
both need to be modified, and the Catalina library must be recompiled.
8. The WiFi definitions have been removed from the P2_MASTER and P2_SLAVE
platform configuration files (i.e. P2MASTER.inc and P2SLAVE.inc) since
this interfered with the ALOHA serial functionality. Instead, two new
platform configurations have been added - P2_WIFI_MASTER and P2_WIFI_SLAVE
(i.e. P2WIFI_M.inc and P2_WIFI_S.inc) which can be used if both WiFi RPC
and ALOHA serial RPC capabilities are required. If only the WiFi RPC
capabilities are required, P2_WIFI can still be used.
5. Catalyst now includes a new Lua script (script.lua) that can be used to
invoke any Catalyst script from the command line just by typing the script
name, without having to say 'exec script'. See script.lua for more details.
Other Changes
-------------
1. A significant bug in the implementation of getenv() has been fixed. The bug
affected only Propeller 2 programs that used getenv(), and Lua programs
(because Lua uses getenv() during initialization), and it also generally
only affected XMM programs. The bug caused memory corruption, usually on
startup. Programs that started ok generally ran properly thereafter. A
previous workaround to this bug was to add calls to _align_sbrk() to move
the C heap to another location - this generally worked around the problem
but didn't solve it. These calls have now been removed, but the
_align_sbrk() function is still available for other purposes. Affected the
Propeller 2 only.
I will support this release, but now that Catalina has all the building blocks I needed, I will be taking a break from Catalina development to do some real software development for a change
Catalina 8.5 contains an error in the files demos/eLua/example/COMMON.LUA and demos/eLua/hybrid/COMMON.LUA
Both files contain an incorrect definition of the invoke function. The correct definition is:
function invoke(f, x)
return bs.deserializeN(svc.serial(INVOKE_SVC, bs.serialize(f, x), 100), 1)
end
Github has been updated. SourceForge users should edit the files manually (NOTE: don't just edit one and copy it over the other - the two files have differences).
All users should use the Catalyst command exec rebuild to rebuild the binary versions of the files before executing them.
Comments
Cool, random data is in the buffer now.
Oh, I've found the problem with fwrite() too. I never knew about the distinction between items and bytes. I guess I'd always fluked it in the past. That and Flexspin treats them as the same so I'd accidentally got it wrong there and not known it.
Ouch, could be better.
Also, I'm seeing an odd behaviour with text sent to the terminal. The writing portion oddly buffers the whole line before emitting it. While the verifying and read speed portions both emit in real time.
Just a question of how much stuff is in the buffer at the time, I think. Add an fflush after each putchar to force it out - i.e.
putchar('.'); fflush(stdout);Ross.
Yeah, that works. But it's not needed for the same action during fread(). Which I see as strange.
BTW: The second time through using the bigger file buffer is same behaviour - without fflush() the terminal doesn't emit until end of line for fwrite() portion, while for both fread() portions it emits immediately.
Here's an updated version with more dots during the verifying portion.
I'm thinking fopen() in write mode must be changing something internal to char handling.
What's happening is that fread() automatically flushes all open line-buffered output streams - including stdout - but fwrite() does not.
This appears to be in accordance with the C specification, which says (in part) ...
Ross.
Hehe, very good. It's allowed according the spec. - Which I've never read.
"implementation-defined" means you can choose though. Is there a reason you decided to make reads and writes behave differently to each other?
I would like to say it was a deliberate and conscious decision - but in fact I didn't write that code, I simply ported it for Catalina
The original "stdio" code Catalina uses comes from the Amsterdam Compiler Kit.
Good point. It would be insane to not use a kit.
EDIT: Oh, wow, it's only fread() that auto-flushes. Any other time I have to manually do it, irrespective of fopen() mode or anything else.
That's weird, why would fread() even bother? What's weirder is shouldn't writes be the ones to flush more?
The C standard says it is so that an fread() after an fwrite() that is intended to display a prompt on a terminal will write the prompt before it does read.
I don't get why that needs an exception. Why not the other way around also? I'm suspicious it's probably an ancient backwards compatibility thing.
The other way around would mean that output streams could never be line-buffered
Doing it this way around means writes can be line-buffered except when you do a read, in which case the output is flushed even if it is not a complete line (which is typical in the case of a terminal prompt).
The only thing that is a little unexpected is that doing an fread() on an input stream flushes all line-buffered output streams - and the reason for that is that stdio does not know which stream is being used to output the prompt - e.g. when you read input from stdin you actually need to flush stdout to ensure the prompt will be displayed on the terminal before you do the input.
The behaviour of your program is a consequence of that.
Ross.
Is that a requirement by the spec though, or is it just implementation defined too?
Separately, I would like to be able to switch off the line buffering of stdout. Or, even better, have libc built without line buffering at all.
Interrupts aren't used for transmitting, right? That would be an excuse to have a buffer at least.
It's a requirement of the spec.
Use setbuf() or setvbuf()
The Catalina kernel does not use interrupts unless threading is used.
Ross.
Reading Linux man pages -
setvbuf(stdout, NULL, _IONBF, 0);did the job. It's consistent now.Thank you for the help!
Aha! Ker .... SPLAT!!! ...
I finally found a nasty bug that has been annoying me for some time. It popped its head up nearly every time I was just about to do a release, and every time it did so I found a workaround for it - but I could not find the underlying cause.
It turned out to be in my implementation of getenv(), which is a standard C function that reads environment variables. It only affected Propeller 2 C programs which used environment variables, or Lua programs (because Lua uses environment variables during initialization), and it also mostly affected programs that used XMM, because it could cause cache corruption during program start up - which made it difficult to find, because if the program started at all it generally worked fine.
I have already uploaded the fix to GitHub - attached is a patch for non-GitHub users. The fix will also be included in the next release.
Ross.
Catalina 8.5 has been released on GitHub and SourceForge
This is a full release. It adds eLua support for using the Parallax WiFi module to make Remote Procedure Calls (RPCs) between propellers. Note that demo and test programs may compile for both the Propeller 1 and 2, but they have only currently been tested on a Propeller 2. Also, some of the WiFi functionality requires a firmware update to the Parallax WiFi module (the updated firmware is included in the release).
Here is the relevant extract from the README.TXT
RELEASE 8.5 New Functionality ----------------- 1. Catalina now supports WiFi-based Remote Procedure Calls (RPCs) in addition to ALOHA serial RPCs. These are now implemented alongside the ALOHA protocol in several eLua variants. It is possible to have just ALOHA services, just WiFi services, or a combination of both. All the existing ALOHA example programs have been updated to use either ALOHA or WiFi, and an example of using both in the same program is given in a new 'hybrid' example. See the document 'ALOHA from Lua' for more details. 2. The binaries generated by compiling the ALOHA versions of eLua have been renamed alua, aluax etc to be consistent with the new WiFi RPC versions of eLua (which are called rlua, rluax etc). No changes to functionality. 3. The eLua programs now load a generic 'serial' module if Lua is compiled with either the 2 port or 8 port serial plugin library (i.e. -lserial2 or -lserial8). Lua programs can use the 'serial' module without needing to know if it is using the 2 port serial or 8 port serial plugin. 4. Lua now includes a module ('wifi') that allows access to the C WiFi support functions. The module is loaded automatically if Lua is linked with the 'wifi' library (e.g. compiled with -lwifi). 5. All the pre-compiled versions of the eLua demos have been removed, since they will all need to be modified and recompiled to use the new WiFi RPC capabilities. 6. Base64 encode and decode routines have been added to the Catalina library. See include/base64.h for details. 7. The buffer size of the 8 port serial plugin has been increased from 32 bytes to 1024 bytes to accommodate the new WiFi RPC functionality. Since there are 16 such buffers (one for each direction of each port) this can make programs that use this plugin up to 16k larger. If this causes problems, the files target\p2\s8serial.t and source\lib\serial8\core.c both need to be modified, and the Catalina library must be recompiled. 8. The WiFi definitions have been removed from the P2_MASTER and P2_SLAVE platform configuration files (i.e. P2MASTER.inc and P2SLAVE.inc) since this interfered with the ALOHA serial functionality. Instead, two new platform configurations have been added - P2_WIFI_MASTER and P2_WIFI_SLAVE (i.e. P2WIFI_M.inc and P2_WIFI_S.inc) which can be used if both WiFi RPC and ALOHA serial RPC capabilities are required. If only the WiFi RPC capabilities are required, P2_WIFI can still be used. 5. Catalyst now includes a new Lua script (script.lua) that can be used to invoke any Catalyst script from the command line just by typing the script name, without having to say 'exec script'. See script.lua for more details. Other Changes ------------- 1. A significant bug in the implementation of getenv() has been fixed. The bug affected only Propeller 2 programs that used getenv(), and Lua programs (because Lua uses getenv() during initialization), and it also generally only affected XMM programs. The bug caused memory corruption, usually on startup. Programs that started ok generally ran properly thereafter. A previous workaround to this bug was to add calls to _align_sbrk() to move the C heap to another location - this generally worked around the problem but didn't solve it. These calls have now been removed, but the _align_sbrk() function is still available for other purposes. Affected the Propeller 2 only.I will support this release, but now that Catalina has all the building blocks I needed, I will be taking a break from Catalina development to do some real software development for a change
Ross.
Catalina 8.5 contains an error in the files demos/eLua/example/COMMON.LUA and demos/eLua/hybrid/COMMON.LUA
Both files contain an incorrect definition of the invoke function. The correct definition is:
Github has been updated. SourceForge users should edit the files manually (NOTE: don't just edit one and copy it over the other - the two files have differences).
All users should use the Catalyst command exec rebuild to rebuild the binary versions of the files before executing them.
Ross.