P1 Longing for logging
iseries
Posts: 1,492
in Propeller 1
You have an idea for a project that can measure the voltage and current of a device. Now you need to record that data so you can monitor the results. There are a few ways to do that and I will try to show each of these ways.
1. Use the terminal output to list the data as it happens. This is the simplest and most often used approach but lacks fiscal recording of the data as well as being tied to the desktop. What you want to monitor is most often not next to you.
2. With the Propeller boards that are out there they have external flash memory that holds the program that is loaded when they are powered on. These units come with 64k bytes of flash but the program only uses 32k of the 64k. This leaves 32k of space to write log data to.
3. By adding an external flash memory card you can log to this memory with the added advantage of being able to remove it or swap it out for other memory so that you can be recording data at the same time you are looking at the previous day’s data.
4. Use a Micro SD card to write the data to a file on the card that can later be read by any computer. This is a complex device to write to but fortunately that hard work has already been done so all we need to do is hook it up to the Propeller board.
5. Use a WiFi module to send the data to another device. This has the advantage of being able to see the data and not have to play with a recording device. This is more complex as the recording device must be up and running otherwise you have no data. Also if the recording device fails for some reason you could lock up the sending device. In either case you lose logging data which is not desirable.
First let’s look at a program that demonstrates number 2 above. Recording data to the flash memory built into the Propeller board itself. The program will write 3 pieces of information that we are interested in. Time, Voltage and Current data will be recorded and saved in the log.
To do this I used a DS1302 Date/Time unit and An INA240 voltage current module from Adafruit. I wrote two libraries for these devices so all I needed to do was plug them into the Propeller board. I used the DS1302 to set the tick counter of the Propeller chip so it would have the current date and time information just like your desktop computer does. The INA240 device output digital voltage and current reading so I only needed to get those values and write them to the log. Below is that program.
Now we could have written the actual text version of the date time to the log but this is messy and uses up a lot of log space with no real advantage. So instead I wrote the Tick value which represents the date time. The trick is to read the time value often enough so that it doesn’t lose track of time. The magic number is less than 54 seconds which is the roll over value.
The only other values that need to be written are the voltage and current which is in hundredths of a volt and milliamps. So all the values fit nicely in three integers and in this case long values. So each log entry requires 12 bytes of data, four bytes for each piece. Doing the math says we can record more than 2500 entries.
Now in addition to the recording program I needed to build a display program that would read the flash memory and convert the data back to their original values. Also the program needed to clear the log file so that I can start over and record some new values. Below is that program with the erase function commented out.
Number 4 requires an SD card to log to. But once the logging starts you cannot remove the SD card. There is no unmount function built into the library function. So the only way to change memory cards is to reboot the unit with a different card.
Most of functions are built into the SD card routines that follow standard programming done on other platforms. In this case I will log the same data except in text form. We could write the data in binary format but then we would need a program on the other side to read the data back. Logging the data in text format with commas as separators makes the most sense. Even though the date time is not broken down it can easily be done with some spreadsheet code. Below is that program.
Here is a sample of the data output to the log:
Mike
1. Use the terminal output to list the data as it happens. This is the simplest and most often used approach but lacks fiscal recording of the data as well as being tied to the desktop. What you want to monitor is most often not next to you.
2. With the Propeller boards that are out there they have external flash memory that holds the program that is loaded when they are powered on. These units come with 64k bytes of flash but the program only uses 32k of the 64k. This leaves 32k of space to write log data to.
3. By adding an external flash memory card you can log to this memory with the added advantage of being able to remove it or swap it out for other memory so that you can be recording data at the same time you are looking at the previous day’s data.
4. Use a Micro SD card to write the data to a file on the card that can later be read by any computer. This is a complex device to write to but fortunately that hard work has already been done so all we need to do is hook it up to the Propeller board.
5. Use a WiFi module to send the data to another device. This has the advantage of being able to see the data and not have to play with a recording device. This is more complex as the recording device must be up and running otherwise you have no data. Also if the recording device fails for some reason you could lock up the sending device. In either case you lose logging data which is not desirable.
First let’s look at a program that demonstrates number 2 above. Recording data to the flash memory built into the Propeller board itself. The program will write 3 pieces of information that we are interested in. Time, Voltage and Current data will be recorded and saved in the log.
To do this I used a DS1302 Date/Time unit and An INA240 voltage current module from Adafruit. I wrote two libraries for these devices so all I needed to do was plug them into the Propeller board. I used the DS1302 to set the tick counter of the Propeller chip so it would have the current date and time information just like your desktop computer does. The INA240 device output digital voltage and current reading so I only needed to get those values and write them to the log. Below is that program.
#include "ina260.h"
#include "simpletools.h"
#include "ds1302.h"
#include <stdlib.h>
#include <time.h>
void Log(int, int, int);
#define MEMSCL 28
#define MEMSDA 29
#define MEMADR 0x50
#define MEMEND 0x8000
#define INACLK 16
#define INADTA 17
#define TIMMISO 8
#define TIMMOSI 9
#define TIMSCLK 10
#define TIMCS 11
struct logptr
{
short magic;
short last;
} LogPtr;
i2c mem;
time_t t;
int i;
int main()
{
int Current, Voltage;
int Ticks;
putenv("TZ=CST6CDT");
DS1302_open(TIMMISO, TIMCS, TIMSCLK, TIMMOSI);
DS1302_setDateTime();
i2c_open(&mem, MEMSCL, MEMSDA, 1);
i = INA260_open(INACLK, INADTA);
if (i != 0x5449)
{
printi("No INA260 Device\n");
while (1)
pause(1000);
}
INA260_setConfig(INA260_BOTH_CONTINUOUS, INA260_1100, INA260_1100, INA260_AVG1024, 0);
while(1)
{
Current = INA260_getCurrent();
Voltage = INA260_getVoltage();
Ticks = time(&t);
printi("%d,%d,%d\n", Ticks, Voltage, Current);
Log(Ticks, Voltage, Current);
i = time(&t) + 900;
while (time(&t) < i)
pause(1000);
}
}
void Log(int t, int v, int c)
{
char data[12];
int Last;
i2c_in(&mem, MEMADR, MEMEND, 2, (unsigned char*)&LogPtr, sizeof(LogPtr));
if (LogPtr.magic != 21)
{
Last = sizeof(LogPtr);
LogPtr.magic = 21;
}
else
Last = LogPtr.last;
memcpy(data, &t, 4);
memcpy(data+4, &v, 4);
memcpy(data+8, &c, 4);
if (Last < 0)
return;
i2c_out(&mem, MEMADR, MEMEND + Last, 2, data, 12);
LogPtr.last = Last + 12;
Last = 0;
while (i2c_busy(&mem, MEMADR) != 0)
{
if (Last++ > 5)
return;
pause(1);
}
i2c_out(&mem, MEMADR, MEMEND, 2, (unsigned char*)&LogPtr, sizeof(LogPtr));
}
The program defines a structure with two elements, magic and last. The magic is a special value that is written so that it knows that log data exists. The last element tells the program where to write the next piece of information to the log.Now we could have written the actual text version of the date time to the log but this is messy and uses up a lot of log space with no real advantage. So instead I wrote the Tick value which represents the date time. The trick is to read the time value often enough so that it doesn’t lose track of time. The magic number is less than 54 seconds which is the roll over value.
The only other values that need to be written are the voltage and current which is in hundredths of a volt and milliamps. So all the values fit nicely in three integers and in this case long values. So each log entry requires 12 bytes of data, four bytes for each piece. Doing the math says we can record more than 2500 entries.
Now in addition to the recording program I needed to build a display program that would read the flash memory and convert the data back to their original values. Also the program needed to clear the log file so that I can start over and record some new values. Below is that program with the erase function commented out.
#include "simpletools.h"
#include "simplei2c.h"
#include <stdlib.h>
#include <time.h>
#define XSCL 28
#define XSDA 29
#define MEMADR 0x50
#define MEMEND 0x8000
struct logptr
{
short magic;
short last;
} LogPtr;
char Buf[12];
i2c mem;
time_t t;
struct tm *tmInfo;
int i;
int s, e;
int C, V;
int main()
{
putenv("TZ=CST6CDT");
i2c_open(&mem, XSCL, XSDA, 1);
printi("Starting\n");
i2c_in(&mem, MEMADR, MEMEND, 2, (unsigned char*)&LogPtr, sizeof(LogPtr));
if (LogPtr.magic != 21)
{
printi("No Log Data\n");
exit(1);
}
s = sizeof(LogPtr);
e = LogPtr.last;
printi("s: %d, e: %d\n", s, e);
while (s < e)
{
i2c_in(&mem, MEMADR, MEMEND + s, 2, Buf, 12);
memcpy(&t, Buf, 4);
memcpy(&V, Buf+4, 4);
memcpy(&C, Buf+8, 4);
tmInfo = localtime(&t);
printi("%04d/%02d/%02d %02d:%02d:%02d,%d,%d\n", tmInfo->tm_year+1900, tmInfo->tm_mon+1,
tmInfo->tm_mday, tmInfo->tm_hour, tmInfo->tm_min, tmInfo->tm_sec, V, C);
s += 12;
}
//i2c_out(&mem, MEMADR, MEMEND, 2, Buf, 2);
exit(0);
}
Number 4 requires an SD card to log to. But once the logging starts you cannot remove the SD card. There is no unmount function built into the library function. So the only way to change memory cards is to reboot the unit with a different card.
Most of functions are built into the SD card routines that follow standard programming done on other platforms. In this case I will log the same data except in text form. We could write the data in binary format but then we would need a program on the other side to read the data back. Logging the data in text format with commas as separators makes the most sense. Even though the date time is not broken down it can easily be done with some spreadsheet code. Below is that program.
#include "ina260.h"
#include "ds1302.h"
#include "simpletools.h"
#include <stdlib.h>
#include <time.h>
void Log(int, int, int);
#define INACLK 16
#define INADTA 17
#define TIMMISO 8
#define TIMMOSI 9
#define TIMSCLK 10
#define TIMCS 11
#define SDDO 2
#define SDCLK 3
#define SDDI 4
#define SDCS 5
static volatile int Sd;
char LogName[] = "Log.txt";
FILE *fp;
time_t t;
struct tm *timeinfo;
char Data[128];
int i;
int main()
{
int Current, Voltage;
int Ticks;
putenv("TZ=CST6CDT");
DS1302_open(TIMMISO, TIMCS, TIMSCLK, TIMMOSI);
DS1302_setDateTime();
time(&t);
timeinfo = localtime(&t);
dfs_setDefaultFileDateTime(timeinfo);
Sd = sd_mount(SDDO, SDCLK, SDDI, SDCS);
if (Sd != 0)
{
printi("No SD Card (%d)\n", Sd);
exit(1);
}
i = INA260_open(INACLK, INADTA);
if (i != 0x5449)
{
printi("No INA260 Device\n");
while (1)
pause(1000);
}
INA260_setConfig(INA260_BOTH_CONTINUOUS, INA260_1100, INA260_1100, INA260_AVG1, 0);
while(1)
{
Current = INA260_getCurrent();
Voltage = INA260_getVoltage();
Ticks = time(&t);
printi("%d,%d,%d\n", Ticks, Voltage, Current);
Log(Ticks, Voltage, Current);
i = 0;
while (i++ < 900)
pause(1000);
}
}
void Log(int t, int v, int c)
{
int len;
if (Sd == 0)
{
len = sprinti(Data, "%d,%d,%d\r\n", t, v, c);
fp = fopen(LogName, "a+");
fwrite(Data, 1, len, fp);
fclose(fp);
}
}
Here is a sample of the data output to the log:
1582719325,753,22 1582719335,753,26 1582719345,753,26 1582719355,753,26 1582719365,753,26 1582719375,753,26 1582719385,753,26 1582719395,753,26 1582719405,753,26 1582719415,753,26 1582719425,753,26
Mike
3264 x 2448 - 2M
Comments
Notes:
Not all propellor boards have 64KB EEPROM (yes it’s EEPROM not Flash on the P1) although most do.
On the SD card, it’s safer to have a big log file predefined and overwrite that file. This way, if the file doesn’t get closed due to a power fail, the data is not lost. I didn’t look at the code to see what you are doing.
{"Date":1582746930, "Voltage":1185, "Current":48} {"Date":1582746940, "Voltage":1185, "Current":48} {"Date":1582746951, "Voltage":1185, "Current":53} {"Date":1582746961, "Voltage":1185, "Current":53} {"Date":1582746972, "Voltage":1182, "Current":98} {"Date":1582746982, "Voltage":1184, "Current":58}The program uses the Parallax WiFi module with updated firmware to allow using UDP. With UDP setup to the broadcast address allows the program to publish the information across the internal network and be picked up by anyone that is listening for that information. In this case the PORT number determines how to listen for that data coming from the Wifi board. Below is the program that builds and sends the JSON data.#include "ds1302.h" #include "ina260.h" #include "esp8266.h" #include "simpletools.h" #include "fdserial.h" #include <stdlib.h> #include <time.h> #include "JSON.h" void Log(int, int, int); char *ConvertInt(int); #define ESPTx 4 #define ESPRx 3 #define INACLK 16 #define INADTA 17 #define TIMMISO 8 #define TIMMOSI 9 #define TIMSCLK 10 #define TIMCS 11 fdserial *fd; char url[] = "101.1.1.255"; char Buffer[128]; char D[16]; time_t t; int i; int handle; int main() { int Current, Voltage; int Ticks; putenv("TZ=CST6CDT"); DS1302_open(TIMMISO, TIMCS, TIMSCLK, TIMMOSI); DS1302_setDateTime(); i = INA260_open(INACLK, INADTA); if (i != 0x5449) { printi("No INA260 Device\n"); while (1) pause(1000); } INA260_setConfig(INA260_BOTH_CONTINUOUS, INA260_1100, INA260_1100, INA260_AVG1024, 0); fd = esp8266_open(ESPRx, ESPTx); handle = esp8266_udp(url, 0x2359); if (handle < 0) { printi("Failed! %d\n", i); while(1) pause(1000); } // give time for wifi to connect pause(5000); while(1) { Current = INA260_getCurrent(); Voltage = INA260_getVoltage(); Ticks = time(&t); printi("%d,%d,%d\n", Ticks, Voltage, Current); Log(Ticks, Voltage, Current); i = time(&t) + 900; while (time(&t) < i) pause(1000); } } void Log(int t, int v, int c) { memset(Buffer, 0, sizeof(Buffer)); json_init(Buffer); json_putDec("Date", ConvertInt(t)); json_putDec("Voltage", ConvertInt(v)); json_putDec("Current", ConvertInt(c)); esp8266_send(handle, Buffer); } char *ConvertInt(int value) { sprinti(D, "%d", value); return D; }Then using a C# program I was able to listen for these packets of JSON data to log and display a graph of the data. A sample is attached.Mike
Yes, loss of data is accounted for by opening the file writing the data and then closing it.
Mike
One of my original P1 products was a data logger module that I programmed in Spin and used FSRW but this was limited to standard SD cards and FAT16. Later on I used Tachyon with data logger products that also incorporated 10/100 Ethernet and FTP protocols, but I used preallocated FAT32 files so all I had to do was write the data with a terminator and flush the 512 byte block. For all practical purposes the card could be removed at anytime (even though there is a very very small chance that the latest block write might be interrupted). There was no need to "close" the file since the directory information was not being updated, which was also on purpose since continual updating of FAT and directory entries could eventually kill your Flash prematurely.
Remember that although the SD card presents the Flash as blocks of 512 bytes, the actual Flash pages can be very large and every time you write to the directory it needs to erase and wear-level a very large chunk of Flash. Eventually everything is wear leveled down to a low level of reliability so it's not a good idea to treat an SD card the same as you would a hard drive by updating the directory every second or continually.
Yes, but most people want to be able to read the SD card in any PC, hence the common FAT32. However, there is no problem with FAT32 as long as you don't touch that part of it and simply work with the blocks within the file. A file system within a preallocated file. Even littlefs still can't do anything about card removal in the middle of the SD card internal controller writing a block to Flash. (actually, it is erasing the large page that takes time, the actual internal block write is fast).
stamps instead of a real time clock? The packets could represent
1/2 sec, 1 sec, 2 sec, .., 5 sec, and the time increment will determined
before recording the data to an eeprom.
The data acquisition unit is for a racing engine, and inside the engine
compartment of a (hot and humid) JetSki hull. It and consists of 4-type K
thermocouples (2 cht's, 2 egt's), tachometer, BME 280 temp, pressure,
humidity sensor, and I would like to record the current ignition timing.
All # packets sensor data and ignition timing will be logged to only an
eeprom to prevent problems caused by vibration and humidity.
Do I devote a whole cog to just performing tasks? and writing to the
eeprom? Not all of the data needs to be recorded in the same intervals.
Bill M.
I would put devices in cogs that are taking readings and have the main loop pull them in and log them for you.
Mike
Hi Peter, I was looking for this app different times but couldn't find it. Does it run using the latest Tachyon?
The software modules are there but the apps were specialized and and destined for "high security" uses, although generally I try to keep as much of a project open as I can. I originally had a lot of problems getting this to work with V3 because I kept running out of memory supporting the FAT32 SD and the WIZnet with FTP and HTTP protocols. Then by using wordcode instead of bytecode with V4 I suddenly had room to spare. I will see what I can dig up and reuse. Now V5 is overall faster and uses less memory.
You mention "The program uses the Parallax WiFi module with updated firmware to allow using UDP". Is this updated firmware generally available or not?
For extended deployments, the logger has to know what to do when it reaches the end of memory, to stop, or to wrap around. And and for repeated deployments, there should be some degree of wear leveling, so, rather than starting at the bottom of memory on each run, it should start each new run where the last one left off or some such expedient. All that requires management of several pointers and modular math. I like to keep labile pointers in battery backed ram in a clock chip.
The Wifi updated firmware was a personal project of mine to add the udp protocol to the firmware along with hiding the unit from being discovered by the IDE. It also allows you to set the IP address, gateway and DNS server for the unit.
I guess I could give you the OTA file to upload but it is not support by Parallax.
Mike
Understood no parallax support. I would still be interested in what you would be willing to share.
Thanks
Right, I forgot about the page write boundary's. So changing the LogPtr size to 16 (int, int, int, int) and changing the data size to 16 should fix that. So now everything is on a 64k page and all the writes will line up.
Now we can only write about 2,000 log entries and 1 million log entries before the chip is no longer usable.
Mike
Not true at all. Read the datasheet and it will say something like "Minimum", not "Maximum", and at a certain temperature. I think I ran tests that never failed even when I approached 10 million writes in the same location.However you wouldn't be writing all those log entries to the same location, they would advance and be spread over the whole chip. The wear applies to the automatic erasing before writing part which is at the page level and on the larger 128kB chips the page size is 256 bytes. So each page will be erased and rewritten internally each time a 16 byte entry is written which means 16 times before it moves on to the next page. Then by the end of writing to the whole chip you will have only reduce the lifetime by 16 writes, not 8,192. After you have filled the chip 62,500 times you will have approached the "minimum" endurance. Failure by definition in this case is when even just a single bit can no longer be programmed reliably, then the entire chip is deemed to have failed.
For sure, I will be implementing a new version that won't be tied to any particular end user. Before the P1 even I've had SD data loggers based on ARM chips that are still going strong, but the later P1 versions allowed that data to be collected over the Internet.
If you want to want to use a record size that is not a power of two, you can do something like the following. (In Spin) The PUB peels off and writes as many bytes as will fit on the first page, and then writes the rest to next page. For reading bytes, the eeprom automatically crosses the page boundaries, except for 64kB and chip-select boundaries.
PUB WriteNstring(myEEadrs,myRAMadrs, byteCount) | n ' 24LC256 is 64 byte page, 24LC512 is 128 byte page, AT24CM01 is 256 byte page repeat while byteCount devID := myEEadrs >> 16 << 1 | $A0 ' for >64kB memory, need to merge 17th 18th 19th bit into devID n := (pageSize - (myEEadrs // pageSize)) <# byteCount. ' peel off the bytes that fit i2c.WritePage(devID, myEEadrs|negx, myRAMadrs, n) ' write them, negx commands 16 bit address myEEadrs += n myRamAdrs += n byteCount -= n waitcnt(clkfreq/200+cnt) ' 5ms delay typical (could test)