multicog data retrieval issue.

synapse

All,
ok I am nearing the completion of a project but I have come to in interesting problem and I want to present a few solutions and ask first if these would work first and second if there is a better way to solve the problem?

Project Background:
I have a main board with a 64 line bus extending the capability of the primary board potentially infinitely if handled correctly. ok maybe not infinate but quite a bit. This main board has a primary Propeller that will handle 4 primary functions.

1. MicroSD data Logging
2. XBee Wireless connectivity
3. FDS to secondary Propeller
4. customizable data based activity

The secondary Propeller establishes connectivity for up to 6 additional option boards via 4 pin serial bus with reset and active pin, all of this programming is complete.

This whole system is a stackable add on computer with a Power Supply board and an sd/usb connection.

The Problem:
Customizable Data based Activity on the primary propeller is the problem. well an issue related to it. I have a communications protocol established and a data storage format handled as well as retransmitting the data over the XBee. the problem lies in RAM. I need to access across several cogs a 3 Dimensional array of data that I can't predetermine the size of the second and 3rd dimensions until runtime. A collection would be great. No collections in SPIN that I know of.

Proposed Solution Ideas
This solution involved creating a collection object that runs in it's own cog. as I receive data from a board I can call a method on it setting a variable value with a key. then have a second method that I can retrieve a value. the problem With this solution is I need access to this collection from a third and since I haven't found a way to fire an event from one cog to another. wait. hmm ok I am going to think out loud here. In pre .net windows programming we have something called the message loop. Ok so we still have it it is just hidden. what if in each of my objects I create an interface of sorts. This interface will have a method that when called could tell the main loop to perform some type of action like call a method on another object like requesting an event be fired. How could this work dynamically. Am I making this too complex. is there a simpler way of firing events or requesting dynamic data updates. or a way of telling another object that a value has changed without events and having to check each value every loop.

Mike Green

Do you really need a separate cog to handle the collection? You can make a case for yes and for no. Remember that cogs and objects are orthogonal concepts (independent).

Whatever is handling the collection can either set a flag indicating that the collection has changed or the handler can (at a cost) keep a list of entries that have changed.

If you want an event-driven system, you have to build the event handling into it. That's true for hardware. That's true for operating systems. That's true for other software. You don't get it for free.

Remember that the Propeller only has 32K of main memory and that has to hold any Spin program interpretive code. If you write some of your code in assembly language, that gets loaded into the 2K of cog memory and you can (with some difficulty) reuse the space occupied by the copy in main memory used to load up the cog. In other words ... you're not going to have large collections (or arrays) without using some kind of external memory ... and any external memory is indeed external ... it looks like an I/O device to the Propeller, not an extension of main memory.

synapse

I don't have to have it in a separate cog.

How about this do you think it would be too slow using FSRW with my SD storage to have a file holding the values that I read and write to the only issue is I am using it to log the data in a streaming fashion. If i do this I will have to close the file every read and write to this file. then open the other file and stream the data from a buffer which I think will have overflowed by that point in time.

I guess I could limit the number of variables that can be checked for actions. unlimited data logging and say 32 hard coded longs that can have decisions based on their values. I would say in most cases only 10 values at most would have any real time needs.

Mike Green

1) I think you are getting caught up in details of implementation way too early in the design process. This is a common pitfall that's often solved by prototyping the program. Don't decide on hard limits too early and don't limit the capabilities of the program too soon. You may find that you can do more and better than you think. On the other hand, you may find that your desires are vastly greater than the capabilities of the hardware. Do a little experimenting.

2) Kye's SD card driver (here) allows multiple files to be opened at the same time. Remember that a major determinant of SD card speed is the card itself. Different models and manufacturers can have widely different throughputs.

synapse

Thanks Mike I didn't know about that object.

I agree with you on the experimenting and normally I would do that however I have some hard deadlines I have set for my self on the first version due to my current employment status. I have several customers lined up for this system. I have to start making some money real soon so I thought I would try a shortcut and see if anyone else had done this before. I had gotten laid off in October and have spent the last several months looking for a job and working on this project. The job search is about a waste of time there is nothing in my area right now. ok sorry for my rant. I plan on release a working solution then making the firmware better as the product grows. I like the methodology of make it work then make it better. Initially I don't need very much in the way of array access I have a total of 22 longs. but the problem is the expansion of my base set of boards is at the heart of the design. My selling point is that I have great expandability. Maybe I get it out with firmware that supports what is out now then release a new firmware in a few months that exposes the scalability of the system. My concern is I don't want to get too far down that path and realize I needed to have extra memory or I should have went with something like a netduino mini for the main processing. I just don't want to add more weight and cost to my system if the prop can do it.

ok sorry I didn't need to say all that I just don't want to send my boards to prototype until I can prove it out in a breadboard. I just don't have the funds to make a mistake.

Mike Green

You're going to have to describe your system in much greater detail if you want better information. How do you figure throughput? How do you figure how much storage you need? All sorts of stuff has been done with the Propeller from datalogging systems to retro-computing emulators like a Z80 or 6502. Most of the emulators use some external parallel SRAM to increase emulated memory to 64K or more. Some people have built cached emulators to use slower serial SRAM. There are multi-Propeller systems, mostly for emulation use or for high-resolution graphics where the additional memory is needed and one Propeller handles the memory intensive task using most of its I/O pins for the parallel SRAM while the other Propeller handles I/O.

If you haven't looked at it already, you might look at Catalina, a free C compiler for the Propeller. It produces a variety of different kinds of code, each trading off speed for size of program including automatic use of external memory.

MagIO2

So, you say it's a 3 dimensional array with runtime dependent sizes in 2 dimensions. But can you define a maximum size? Then you could create the array in advance. Either it's small enough, so that it fits into your free RAM-max. needed stack-size or you can create a file of that size.
With a max. size the address calculation is fixed:
address := x*maxx + y*maxy + z @startofarray
(for bytes)

I just want to bring in my virtual memory ;o) It's only a set of functions to be added to the FSRW. It allows to have an open file for read/write access with the usual FSRW-functions. Besides that you have access to one or up to four swap-files of any size. And you can access any memory location inside of this file. Only some changes would be needed if you want to access the same 512 byte-block from several COGs.

Here is the link:
http://forums.parallax.com/showthread.php?117573-Virtual-Memory