QS board + RPi = data logging
Rsadeika
Posts: 3,824
I have a prototype setup of a QS board concoction, which includes a sht11 and an XBee. This is setup outside in a weather protected place which runs 24/7 that provides instant access for real time temperature and humidity readings. On the RPi I have a Python program that requests the data from the QS, and manually logs it to a file. Could not figure out how to get Python to do threading or multiprocessing to suit my needs.
This is a question for the Linux people, what is the best program, available on the RPi, to take the contents of the datalog file, and prepare it for data extraction to create a graph(s) of some sort that could be viewed on different OS machines? The datalog file would contain a typical line that would look like this: '07/26/15,09:30:30,Temperature 77 F, Humidity 67 %;'.
The time is based on a 24 hour clock, and the file access is done in append mode, so I could end up with one very large datalog file, I guess that needs to be addressed also. I guess first, how do I create some sort of meaningful graph of the datalog contents?
Ray
This is a question for the Linux people, what is the best program, available on the RPi, to take the contents of the datalog file, and prepare it for data extraction to create a graph(s) of some sort that could be viewed on different OS machines? The datalog file would contain a typical line that would look like this: '07/26/15,09:30:30,Temperature 77 F, Humidity 67 %;'.
The time is based on a 24 hour clock, and the file access is done in append mode, so I could end up with one very large datalog file, I guess that needs to be addressed also. I guess first, how do I create some sort of meaningful graph of the datalog contents?
Ray
Comments
I'm not sure I'm with your requirement there. As you have a program up and running on the Pi that collects the data from your QS why not extend that and have it also transform the data into whatever format you want? Whilst you are it it could also send the data out to some remote machine.
I'd be wanting to send it to a remote web server using an HTTP POST request. That server could also serve up a page that displays a chart of the data.
I advise against writing to a file on the SD card all the time. That can cause file system corruption issues when the power gets cut.
Since this all still in the experimental stage, I am trying different approaches to solve a problem. I think that the RPi will be great for data collection, but it might have a resource problem when it comes to managing the accumulated data.
So, I guess the next step is to figure out a nifty way of moving the datalog file off of the thumb drive to a remote machine, automatically. I do have a Widows 8.1 box that is acting as a pseudo server of sorts, I guess I could use that as my remote machine.
I know nothing about remote web servers, although that seems like an interesting idea. But, not sure if I want to go into that direction at this time, sounds to complicated for my experiment.
Ray
I think your time and date format should be okay as it is. The spreadsheet programs will allow you to specify which sort of data each column contains. Your date and time stamps should be recognized as such.
You could import the data with the "Temperature" and "Humidity" left in place but this seems like a waste of space. I know OpenOffice allows one to use a space as a delimiter you could strip out the extra words as you import the data to a spreadsheet. It seems like it would be better to just record the numbers with comma delimiters.
If you save the file with this sort of data as a .csv or .txt file, just about any spreadsheet program should be able to open it and process it into graphs and charts.
On my Windows 7 desktop, I now have LibreOffice upgraded to 4.4, and the graphs part, with LibreCalc, seems to be working as expected. I figured out what values the program likes, and that is what the Python program puts out.
The experiment with LibreCalc was a good indicator that I probably should switch to LibreBase. Now I have to figure out how to create a base file that LibreBase will like. This will probably be a little more difficult than just working with LibreCalc.
I tried experimenting with an FTP method of transferring files, between my Windows and the RPi, no success. On the RPi , from the command line, when I type in 'ftp' all I get is a 'bad command' response. And I did install the latest suggested ftp program on the RPi, just does not want to work from the command line.
Ray
You could set up the ftp server elsewhere, like your desktop, and use Python to handle the client side of things.
Note: edited for spelling , grammar, and lack of caffeine...
I have the Python program writing the accumulated data to a csv file on the thumb drive. I will create a file for a month of collection, the month of collection will be used by LibreCalc, at which time I can create a rough graph of the months data.
After I am done with the LibreCalc part, I can then copy that data to a LibreBase setup, where the data will be permanently stored. I think with LibreBase I can then setup queries that will give me more comparison options.
At this time I only have one Propeller/sht11 data acquisition unit, which is outside the house. I will be adding another Propeller/sht11 data acquisition unit in an indoor environment, and that acquired data will eventually be added to the established LibreBase setup. Hopefully LibreOffice can keep up with my experiments.
Ray
Switching over to a hardline access would really be a major redesign, which would probably include having an RPi in the mix so I could get to accessing the data via the LAN. Not sure of any other solution that would work?
Ray