PDA

View Full Version : Proposed Clock format for propeller programs...



Cluso99
03-26-2012, 01:14 AM
General 32bit Date/Time format proposal for propeller programs...

A modified FAT16/32 Date/Time format...


Year20xx Month Date Hours Minutes Seconds
(00-63) (1-12) (1-31) (00-23) (00-59) (00-59)
000000___0000____00000___00000____000000____000000

Above uses year base 2000 providing up to year 2063, and accurate to the second.
We can update the whole 32bit value at once, so no possibility of reading between a partial update.


Conversion to FAT16/32 file format is simple. Shift >>1 then add 16<<25
(i.e. Shift right 1 place, then add 16 to the year because base is 1984 instead of 2000)

Year1984+ Month Date Hours Minutes Seconds*2
(00-127) (1-12) (1-31) (00-23) (00-59) (00-58)
0000000___0000____00000___00000____000000____00000

Above FAT16/32, the seconds are only counted every 2 seconds because it is 1 bit short!
This gives years 1984-2111, but note 2100 is not a leap year. The time is held in 1 16bit value and the date in another 16 bit value.


FYI: seconds in 4yrs = 126,230,400. So unsigned 32bits can hold 132 years.

From 1904 to 2096, every 4 years are leap years. So, no need to worry about 100 year leap years.

If we also hold a 32bit value of seconds, I suggest we use a base of year 2000. I am unsure if this is necessary as well as the date/time value.


1. It is quite likely that Kye's SD driver could be modified to maintain the clock value in hub between SD usage (Dumb RTC using prop). Lonesock does this in his fsrw routines (using just a seconds counter in hub).

2. If a real RTC is used, then an object could be used to read and set a real RTC clock chip, and update the SD RTC counter value(s). Or it could be done via a separate cog and object.

3. Any program could obtain the seconds/time at any time by simply reading a live hub location. No need for a request because it will always be updated by the RTC cog in the background.

Any ideas or comments on...

* The format ??
* The best way to implement this??


P.S. I have asked similar questions relevant to the Catalina implementation. It is also relevant to my OS and others. I hope this does not offend the moderators for asking similar questions on other respective threads.

cavelamb
03-26-2012, 02:03 AM
General 32bit Date/Time format proposal for propeller programs...

A modified FAT16/32 Date/Time format...


Year20xx Month Date Hours Minutes Seconds
(00-63) (1-12) (1-31) (00-23) (00-59) (00-59)
000000___0000____00000___00000____000000____000000

Above uses year base 2000 providing up to year 2063, and accurate to the second.
We can update the whole 32bit value at once, so no possibility of reading between a partial update.


Conversion to FAT16/32 file format is simple. Shift >>1 then add 16<<25
(i.e. Shift right 1 place, then add 16 to the year because base is 1984 instead of 2000)

Year1984+ Month Date Hours Minutes Seconds*2
(00-127) (1-12) (1-31) (00-23) (00-59) (00-58)
0000000___0000____00000___00000____000000____00000

Above FAT16/32, the seconds are only counted every 2 seconds because it is 1 bit short!
This gives years 1984-2111, but note 2100 is not a leap year. The time is held in 1 16bit value and the date in another 16 bit value.


FYI: seconds in 4yrs = 126,230,400. So unsigned 32bits can hold 132 years.

From 1904 to 2096, every 4 years are leap years. So, no need to worry about 100 year leap years.

If we also hold a 32bit value of seconds, I suggest we use a base of year 2000. I am unsure if this is necessary as well as the date/time value.


1. It is quite likely that Kye's SD driver could be modified to maintain the clock value in hub between SD usage (Dumb RTC using prop). Lonesock does this in his fsrw routines (using just a seconds counter in hub).

2. If a real RTC is used, then an object could be used to read and set a real RTC clock chip, and update the SD RTC counter value(s). Or it could be done via a separate cog and object.

3. Any program could obtain the seconds/time at any time by simply reading a live hub location. No need for a request because it will always be updated by the RTC cog in the background.

Any ideas or comments on...

* The format ??
* The best way to implement this??


P.S. I have asked similar questions relevant to the Catalina implementation. It is also relevant to my OS and others. I hope this does not offend the moderators for asking similar questions on other respective threads.

Phil Pilgrim (PhiPi)
03-26-2012, 03:16 AM
Why not use a standard that already exists? Unux time (http://en.wikipedia.org/wiki/Unix_time) is a signed 32-bit integer representing the number of seconds since 0:00:00 on 1 January 1970 UTC.

-Phil

kwinn
03-26-2012, 07:38 AM
Sorry Cluso but I have to agree with Phil on this one. Lets stick with a current standard that is already in reasonably widespread use.

MagIO2
03-26-2012, 09:35 AM
Hmmm ... the wiki-page was a nice read. But as it says there: "This has resulted in considerable complexity in Unix implementations, and in the Network Time Protocol, to execute steps in the Unix time number whenever leap seconds occur."
You really want to do that in the propeller?

I would tend to keep it as simple as possible, which is: If you need a full date-string, use an I2C RTC attached to the EEPROM-pins and read it at least once a day to get the date-part freshly - which then includes leap year changes correctly. If you need the date very often - like when you need periodic logging - I think it makes sense to convert it into the format proposed by Cluso99, as it's
a) easier to update - see the quoted statement above
b) easier to convert to a string
c) gives us another 25 years on top of unix time until we have to switch to 64 bit ;o)

Cluso99
03-26-2012, 10:30 AM
I have spent a lot of time researching this. I have now coded a spin program to keep the time in this format. The advantage is that

1. It is almost totally usable with the fat format... 1 shift and 1 add. We should use this in the os.

2. It has 1 sec accuracy where fat is only 2 sec. This was most likely because of word alignment that we do not require in the prop.

By using a base that is a leap year, it is simpler to calculate. I re fer to unix here.

Don't forget we ned to convert to/from fat time for file updates.

MagIO2
03-26-2012, 12:11 PM
Why would you implement something for leap-year calculation? In case you want to have a timestamp which has the real date - for whatever reason - you have four options:
add an RTC
add a RF-time receiver
add a network-time request
or have a "never switch off-system"
.. well ... you also have the possibility to ask for the date/time with each switch-on ;o)

For me option 4 and 5 are most unlikely! And all other 3 already do the handling of leap-years. You have to read their date anyways if you switch on the propeller! So, re-read at 00:00 is enough to deal with the leap-year problem.

Cluso99
03-26-2012, 02:21 PM
Here is a working version. Note that the program is designed to run in the Propeller OS that I am working on, so you will see I don't use FullDuplexSerial anymore.

There is also a routine to format a string for display purposes. And routines to convert to/from FAT date/time format.

There are routines (commented out and untested) that can be called at least once per second. Well, actually it can be less often, but the seconds will jump. As long as the CNT does not overflow (~50 seconds @ 80MHz or ~40 seconds @ 104MHz).

Next is to add code to initially read from the RTC (I use a DS1340C) or if no RTC, then ask the user.

Here is an extract of some portions of the code...


' Date/Time Format: Year20xx Month Date Hours Minutes Seconds
' (00-63) (1-12) (1-31) (00-23) (00-59) (00-59) values
' 000000___0000____00000___00000____000000____000000
' 3 2 2 2 2 1 1 1 1 0 0 0 \ bits
' 1 6 5 2 1 7 6 2 1 6 5 0 /

PUB CountSeconds | Timer, i
Timer := cnt
date_time := 12<<dt_years_shift | 3<<dt_months_shift | 26<<dt_days_shift {
} | 18<<dt_hours_shift | 44<<dt_minutes_shift | 56
repeat
waitcnt(Timer += clkfreq) ' accurate 1 second delay
UpdateTime
PRI UpdateTime | flag, days, mths
date_time++ ' secs +1
if (date_time & DT_SECONDS_MASK) => 60
date_time += DT_MINUTES_INC ' mins +1
date_time &= !DT_SECONDS_MASK ' secs =0
'next: the shift is on the rhs because it evaluates at compile time so it's faster
if (date_time & DT_MINUTES_MASK) => (60 << DT_MINUTES_SHIFT)
date_time += DT_HOURS_INC ' hour +1
date_time &= !DT_MINUTES_MASK ' mins =0
if (date_time & DT_HOURS_MASK) => (24 << DT_HOURS_SHIFT)
date_time += DT_DAYS_INC ' day +1
date_time &= !DT_HOURS_MASK ' hours =0
days := (date_time & DT_DAYS_MASK ) >> DT_DAYS_SHIFT ' extract the days
mths := (date_time & DT_MONTHS_MASK) >> DT_MONTHS_SHIFT ' extract the months
flag~
if days == 0 '(i.e.=32) ' 32: it is 0 because we only have 5 bits
date_time -= DT_MONTHS_INC ' so remove the rollover to the month
mths-- ' " "
flag~~ ' 32: Jan/Mar/May/Jul/Aug/Oct/Dec: mth+1
elseif days => 31 ' 31: test for 31 days in mth
if mths => 8 ' if mth Aug/Oct/Dec +1 to make odd; Sep/Nov become even
mths++ ' Jan/Mar/May/Jly are already odd
if (mths & 1) == 0 ' now test for an even month which has 30 days
flag~~ ' even= Apr/Jun/Sep/Nov: mth+1
elseif days => 29
if mths == 2 ' February?
if (date_time & DT_LEAPYEAR_MASK) <> 0 ' not leap year?
flag~~ ' not leap year & 29: mth+1
elseif days => 30
flag~~ ' leap year & 30: mth+1
if flag ' inc month?
date_time += DT_MONTHS_INC ' months +1
date_time &= !DT_DAYS_MASK ' days =0
date_time += DT_DAYS_INC ' days +1=1
if (date_time & DT_MONTHS_MASK) => (13 << DT_MONTHS_SHIFT)
date_time += DT_YEARS_INC ' year +1
date_time &= !DT_MONTHS_MASK2 ' months =1 (months already odd)

PRI ConvertToFAT(dt)
Return (dt >> 1) + (16 << 25)
PRI ConvertFromFAT(dt)
Return (dt - (16 << 25)) >> 1


91021

prof_braino
03-26-2012, 03:43 PM
How would this scheme work with "precision logging"?

We had a discussion about logging, and it seems that an RTC is only good to seconds, and we can get very good time sync over the internet, and great time from GPS. (So we're thinking RTC is not going to be a mandatory part for most applications).

The thought was most things would be hooked to the time source that supplies the proper accuracy, IE apps for GPS or internet.

The exceptions would be sort duration apps that are not connected to GPS or internet, these would have to be (manually) sync'd to something external anyway. For these short duration apps, one could just count clock cycles, and use a second cog register or hub word to count overflow. A cog would be good for up to 2,668,799 days, or about 7,311 years. A hub word would be good for 40 days. Being short term, they by definition wouldn't run long enough to drift too much before restart or resync. This would cover 80-90% of of cases.

Based on this, it looked like linux time would simplest. But we didn't commit to any method yet. And this was focused on logging rather than FAT compatibility.

Cluso99
03-26-2012, 04:25 PM
Prof: GPS is not always available indoors because it depends on visibility to the satellites. Its fine on our boat because it is fibreglass, and on cars because they look through the windscreen. GPS is also way more expensive than an RTC. With the prop and an RTC, you can get timing to 1 clock cycle although obviously the instructions take 4 cycles, because you can sync the RTC with the prop which can then count at its xtal frequency.

The internet is no where near as accurate as you cannot determine the latency of the packets received. And of course, this presumes that you have an internet connection to the prop - more expense. So a lot really depends on the apps as to what is required.

What I have is the capability of a cog, or a part of a cogs shared code, to do a soft RTC which may (or may not) have a real RTC in the background to initialise the date/time. One of the circuits I am working on has 3 props, 512KB SRAM, microSD and an RTC, amongst other things. We need to be able to keep (relative) accuracy down to 1 second.

As for the format of the time, I am running a Prop OS on the SD card and FAT16/32 is used. So I already have to deal with the FAT time format. I just did not want to use the exact format because of the 2 second ticks. I may well keep a seconds counter too. If so, I will start it at 00:00:00 on 1Jan2000. BTW I am ignoring any leap seconds.

Mark_T
03-26-2012, 04:54 PM
There are different uses for a clock type, for instance you might be doing carbon-14 dating and want to work tens of 1000's of years into the past, or you might be timing neutrinos flying through the Alps and want nano-second timing.

If you are only really interested in file timestamps then 1 second resulution and a limit of 32 bits might be enough.

If you are doing calculations involving time you prefer a format that is linear - the proposed format is discontinuous.

For general use I don't think this is flexible enough - once you start talking calendars then you may have to represent timezone, deal with historical dates, keep track of the day of the week etc.

I don't think one type is general enough to be "the clock format". Given constraints of memory space and the highly domain-specific applications on microcontrollers people's requirements won't always overlap.

Phil Pilgrim (PhiPi)
03-26-2012, 05:02 PM
Time zones are best handled by always keeping the time internally as UTC and converting only when it has to be displayed or printed out. Otherwise, things become a mess when you move or convert between standard and daylight time.

-Phil

MagIO2
03-26-2012, 07:24 PM
@prof_braino:
Precision logging can easily be done using this proposed time-format plus the cnt-value. You'd simply log the start-time first. With the start-time/cnt you can even correct the time later on if updating the time-format value is to slow, as the seconds are redundant for up to 52 seconds.
After that initial start timestamp all the times you log are as precise as cnt. If it also has to be high-speed, it's of course better to output raw longs.

So, the only question is how precise can you setup the time variable when starting up the propeller. A longwave atomic clock signal would propably be the best if you are close enough to a sender.

pedward
03-27-2012, 01:07 AM
Unix time is the clear winner in my book. The 32bit value is clean up to 2038, when it rolls over. The issues with leap seconds and such are present with any time conversion system. The problem arises that time "repeats" itself, which can happen in any time system.

Unix handles high resolution logging by appending a microsecond resolution timer to the epoch seconds. You can see this easily while using tcpdump. The kernel stamps every packet with a high resolution timestamp, so you get very good resolution. It would be possible to emulate this with the cnt variable, just some clever coding is required to handle overflow.

I would think that once per second you calculate a modulus and offset value to be applied to the system counter, so you get the proper value. I haven't thought it out more than a few moments, so it could be simpler than that.

ersmith
03-27-2012, 01:45 AM
If you use an unsigned 32 bit integer to store a count of seconds since 1970 then the roll-over is delayed until about 2105 or so; that's what the PropGCC library does.

The Posix standard really messed up in its handling of leap seconds (by leaving them out it made calculating elapsed time very difficult, which was kind of silly given that the base "count of seconds" format is otherwise very convenient for elapsed time). It is a widely implemented and used standard, though, so it's a logical choice.

Eric

MagIO2
03-27-2012, 06:11 AM
Cluso99's proposed format is also logical. If you want to have a continuous 32 bit integer you need to have a function called on switch-on which reads the RTC/Nettime/... . All of these will give the time in year/month/day...-format. So, to get to your 32 bit integer you need to implement a calendar which takes care of leap-years, maybe leap-seconds. And for converting back from 32 bit to date-string you need to call the reverse operations for each line you want to log. Output of cluso99's format is pretty easy, no math. Together with cnt update is not even needed each second. Just do it somewhen between 1s and 50s after the last update - just in case you have better things to do. The update itself is also simple ... add second(s), check for overflow, adjust minutes, check for overflow, adjust hours ......

Updating is of course a little more complex especially because you can use a counter for the 32 bit integer solution, but converting to string should be much faster. So, actually it depends on the usage whether one is better or the other.
But this is programming ... it's always about the yin and yang - and you can't have both at the same time ;o)

Cluso99
03-27-2012, 08:59 AM
.....So, actually it depends on the usage whether one is better or the other.
But this is programming ... it's always about the yin and yang - and you can't have both at the same time ;o)

I am always looking to have my cake and eat it too ;)

Actually, I have pretty much decided to keep both my modified FAT version and a seconds version starting with year 2000. I wont be worrying about leap seconds - someone can add a table if they want, but the future leap seconds are not decided, so they cannot be inbuilt anyway. This method will allow 132 years = 2132 but I am not taking 2100 into account as it is really not a leap year. If anyone wants UTC, just add "126,230,400 * 7 + 31,536,00 * 2" = 157,766,400 (30 years in seconds from 1970 to 2000 excluding leap seconds) to the long.

The reason I am using 2000 as the base is because it makes the maths a little easier, and because I need something that uses 2000 as a base for tables. Also it keeps the 2 longs in sync.
Maths is easy... 4 years = 126,230,400 seconds, so keep subtracting this while it is > seconds, accumulating years as +4
Then subtract 31,622,400 for 1 year (leap year), and then 31,557,600 for up to the next 3 years
Once you have the leftover seconds, they will be < 1 year, so now subtract as follows...
- 2,678,400 for Jan
- 2,419,200 for Feb (non leap year) or 2,505,600 (leap year)
- 2,678,400 for Mar
- 2,592,000 for Apr
- 2,678,400 for May
- 2,592,000 for Jun
- 2,678,400 for Jul
- 2,678,400 for Aug
- 2,592,000 for Sep
- 2,678,400 for Oct
- 2,592,000 for Nov
- 2,678,400 for Dec (you will not get here unless you made an error above)
Once you have a smaller remainder than the next months subtraction, subtract as follows...
- 86,400 per day
- 3,600 per hour
- 60 per minute
and your seconds are left (and should be <60)

Dont forget to start you month and day at 1, not 0.
This is a lot quicker using interger maths than divides!!

Because we can update the hub by writing longs, it will not be possible for another cog to read it in the middle of a rollover.

ersmith
03-27-2012, 11:11 AM
Because we can update the hub by writing longs, it will not be possible for another cog to read it in the middle of a rollover.

It is also possible to do this with 64 bit values, as long as all COGs follow the same protocol. If they always write the low word first, followed immediately after by the high word (in the next hub window), and similarly read the low then high word in 2 consecutive hub windows, then all the COGs will see consistent values for the 64 bit value. The only downside to this protocol is that not all high level languages are able to guarantee that the 2 longs will be read in consecutive hub windows, so some PASM glue code may be required.

MagIO2
03-27-2012, 11:55 AM
I would not worry about 32 bit rollover to much in the 64 bit scenario. First get the high 32 bit, then the low 32 bit and then only one check is needed:


if low_32bit == 000000___0001____00001___00000____000000____000000
get high 32 bit again!

because this is the one and only time where you can have a rollover. Even SPIN should be able to do that within one second ;o)

Phil Pilgrim (PhiPi)
03-28-2012, 05:45 AM
In order to foster interest in, and acceptance of, the Unix time standard, and in the spirit of "a bird in the hand" for this thread, I've written an object that converts back and forth between Unix time and date/time data. You can read about it here:


http://forums.parallax.com/showthread.php?138987

-Phil

Cluso99
03-28-2012, 06:21 AM
Nice Phil. I am going to push forward with representing both of my versions based on 2000. It is so easy to convert between the yr2000 seconds anyway.

As for unix, 2032 is now shortsighted. It wasnt back in 1970, and look at how much code is still running. But all is not lost, because by the time we get near 2032 they can use unsigned date and squeeze another 68 years out of it, instead of using -ve to go back from 1970. And we will be on 64 bit processors before you know it anyway.

Phil Pilgrim (PhiPi)
03-28-2012, 06:39 AM
It is so easy to convert between the yr2000 seconds anyway.
Not to flog an apparently dead subject, but have you looked at my Unix time conversion code? It's much more straightforward than I imagined it could be (no ponderous nests of conditionals and exceptions, just integer math). Plus, incrementing Unix time just means adding 1, and computing the difference in times is simply a matter of subtracting.

-Phil

Cluso99
03-28-2012, 10:04 AM
Phil: Just had time to give it a quick look over. Yes, it is way simpler than I originally expected, though there are a few time consuming multiply and divides in there.
The algorithm is the same as calculating Julian Day except of course the base year is 1/1/4713BC.

I am going to run a time test on the conversion and I will post the results.Perhaps I can avoid holding two formats after all. Just holding seconds is way easier if the conversion doesn't take too long.

BTW Nice job on the web docs!