I've wiped all the parity sourced Practrand report files and started the culling and gridding runs afresh.
EDIT: Here's the culled tables, both with and without parity, for comparison. Notably [11 10 12]s parity scores means it shouldn't be included now but I've thrown it in to fill the final score chart.
1. Best xoroshiro32+ triple is [14,2,7].
2. Parity trick makes a huge improvement to bit 0.
3. Bit 1 is much better than bit 0 without parity but weaker than higher bits.
4. Improved bit 0 with parity masks relatively weak bit 1 for [7:0].
I'm most interested currently in which 16-bit subsample of double-iterated xoroshiro32++ [14,2,7,5], i.e. the P2 v1 XORO32 instruction, has the highest PractRand score. Frequency test results below suggest it will be [25:10].
How much better will the top score be than [15:0] or [31:16], I wonder?
Thanks, Evan. Do you think you might have time this week to do 16-bit double iteration tests for [14,2,7,5]? The results would have practical use on the P2 v1 and FPGA implementions. The three scores needed are [15:0], [31:16] and whichever other one is the highest.
How much info are you wanting? Just a few Practrand scores only? I could hand build and score a few individual cases one by one I guess.
What's the chances of doing any more later? If this is likely to recur I should probably work on getting the automation to do it. EDIT: We could grid them all too then.
How much info are you wanting? Just a few Practrand scores only? I could hand build and score a few individual cases one by one I guess.
What's the chances of doing any more later? If this is likely to recur I should probably work on getting the automation to do it. EDIT: We could grid them all too then.
[15:0], [31:16] and [25:10] if individual, just to see the difference. Automated grid to test all 16-bit subsamples (and possibly higher) could be handy, though.
Damn it. There has still been more BCFN glitches getting by undetected. This time at least no need to start over.
Latest scoring logic:
if [ ! -f "$PRreport" ]; then # If no pre-existing Practrand report then run and score the case
"$RNGbin" | stdbuf -o L $PRcommand >"$PRreport"
if [ $? -ne 0 ]; then
printf "Aborted ${RNGbin} - PractRand error\n"
exit 2
fi
fi
# Output the score to console/logfile. This is not score extraction for score table.
extract_score
if [ $bcfncnt -eq $failcnt ]; then # probably incorrect score due to too sensitive BCFN testing
if [ $bcfncnt -gt 2 ];then
printf "Passing because "
else # rerun testing from +1 power
mv "$PRreport" "${PRreport}.tmp"
printf "BCFNs: $bcfncnt, ${RNGbin}, PractRand score: ${scoresize}B - Trying larger ...\n"
scoresizedn=$scoresize
sizeup=$(( $sizekb * 2 ))
"$RNGbin" | stdbuf -o L $PRcommand -tlmin "${sizeup}KB" >"$PRreport"
extract_score
if [ $sizekb -eq $sizeup ] && [ $bcfncnt -gt 0 ]; then # recuring fails
if [ $bcfncnt -lt $failcnt ] || [ $bcfncnt -gt 2 ]; then # initial score was valid, revert
rm "$PRreport"
mv "${PRreport}.tmp" "$PRreport"
printf "BCFNs: $bcfncnt, Fails: $failcnt, Reverted - "
scoresize=$scoresizedn
bcfncnt=0
else # two BCFN glitches in a row! raise another power, rerun again
printf "BCFNs: $bcfncnt, Fails: $failcnt, ${RNGbin} - Larger again ...\n"
sizeup=$(( $sizekb * 2 ))
"$RNGbin" | stdbuf -o L $PRcommand -tlmin "${sizeup}KB" >"$PRreport"
extract_score
if [ $sizekb -eq $sizeup ]; then # plain giving up at three tries, revert
rm "$PRreport"
mv "${PRreport}.tmp" "$PRreport"
printf "BCFNs: $bcfncnt, Fails: $failcnt, Reverted - "
scoresize=$scoresizedn
bcfncnt=0
else # got through it, any more glitches for this case won't be detected on this run
rm "${PRreport}.tmp"
fi
fi
else # revised report is normal, BCFN glitch cleared
rm "${PRreport}.tmp"
if [ $bcfncnt -lt $failcnt ]; then # report the correction
printf "BCFNs: $bcfncnt, Fails: $failcnt, Accepted - "
fi
fi
fi
if [ $bcfncnt -eq $failcnt ]; then
printf "BCFN & Fails: $failcnt, "
fi
fi
printf "${RNGbin}, PractRand score: ${scoresize}B\n"
rm "$RNGbin"
I'm most interested currently in which 16-bit subsample of double-iterated xoroshiro32++ [14,2,7,5], i.e. the P2 v1 XORO32 instruction, has the highest PractRand score. Frequency test results below suggest it will be [25:10].
How much better will the top score be than [15:0] or [31:16], I wonder?
Thanks, Evan. Do you think you might have time this week to do 16-bit double iteration tests for [14,2,7,5]? The results would have practical use on the P2 v1 and FPGA implementions. The three scores needed are [15:0], [31:16] and whichever other one is the highest.
How much info are you wanting? Just a few Practrand scores only? I could hand build and score a few individual cases one by one I guess.
What's the chances of doing any more later? If this is likely to recur I should probably work on getting the automation to do it. EDIT: We could grid them all too then.
[15:0], [31:16] and [25:10] if individual, just to see the difference. Automated grid to test all 16-bit subsamples (and possibly higher) could be handy, though.
Bouncing around the walls again? Hehe. Yeah, I went all in for automatic testing of double iterating the generator. The C source was quick to change although it did require concentration for keeping sorted the separation of word widths. I needed to be fresh for sure.
While working my way through the scripts it started getting messy enough that I decided to do a cleaning round ... and that proved troublesome due to ripple effects right throughout.
Had a day off. Started a functioning culling run today - Looking to be twice the number of passing candidates with 512 MB threshold. Again, none of the [14 2 7 x] candidates made the grade.
I've only just finished the gridding clean-up right now. I'll have a grid done for [14 2 7 5] first ...
I've done the faster half (even-sized apertures) first:
Full 16x16 single iterated grid: Lowest Exponent = 27, Exponent Average = 30.957
Even 8x16 single iterated grid: Lowest Exponent = 27, Exponent Average = 30.242
Even 16x32 double iterated grid: Lowest Exponent = 26, Exponent Average = 31.126
Note the heightened average even though the worst case is lower. And it'll be higher again with the odds included.
At any rate, Practrand scoring has consistently matched the distribution scores all along. I've been a little surprised by how good your numbers are for predictability.
I guess you could say we've done a pretty good job verifying distribution scoring as a rapid candidate selection method. My Practrand based mass scoring approach has been a good test bed for doing the proving but it isn't practical to carry on using for longer worded engines. Although shorter engines would be fine.
I'd like to continue down current track. Finish the double iteration mass scoring. Then also do some testing of XoroshiroXX** as well - Try and get a gauge on the Prop2's free running generator too.
Full 32x32 double iterated grid: Lowest Exponent = 26, Exponent Average = 31.839.
Compared to an average of 30.957 for the single iterated grid scoring, that's not far off average scores being doubled in value. And lowest is half value.
So, based on this one candidate, double iterating destabilises the quality a little ... and distribution likewise?
I'd like to continue down current track. Finish the double iteration mass scoring. Then also do some testing of XoroshiroXX** as well - Try and get a gauge on the Prop2's free running generator too.
Running at about 3 hours per grid (for easy half). Currently at #4 of 53 candidates, so maybe another week to finish the gridding.
Hmm, one thing that has been getting on my wick is the amount of unusable RAM on PCs these days. It crazily increases with total RAM installed. This newish Ryzen setup I've got is losing about 600 MB for no apparent reason. Given the CPU has the whole memory controller built-in, the loses aren't likely to change with motherboards. EDIT: I'd be okay at 1% of that, say 5 MB unusable.
CPU AuthenticAMD, Features Code 178BFBFF, Model Code 00800F11
AMD Ryzen 7 1700X Eight-Core Processor
Measured - Minimum 3992 MHz, Maximum 3992 MHz
get_nprocs() - CPUs 16, Configured CPUs 16
get_phys_pages() and size - RAM Size 31.41 GB, Page Size 4096 Bytes
...
And here's with one DIMM removed. Now the unusable amount is down to about 340 MB. I totally don't get why it's dynamic at all.
CPU AuthenticAMD, Features Code 178BFBFF, Model Code 00800F11
AMD Ryzen 7 1700X Eight-Core Processor
Measured - Minimum 3992 MHz, Maximum 3993 MHz
get_nprocs() - CPUs 16, Configured CPUs 16
get_phys_pages() and size - RAM Size 15.66 GB, Page Size 4096 Bytes
Back to random number testing now. I note me CPU cooler is in need of a dust out too. Packed dust forming into top and bottom fins.
EDIT2: Big Oops! Those Physical RAM sizes weren't actually truly the amount of DRAM addressable. Asking google the right question made a huge difference to my understanding. Turns out those numbers exclude the whole Linux kernel space! And that has a reserved space that sizes as a percentage of the physical RAM installed.
User available (16289976 KB) + Kernel reserved (420332 KB) = Kernel available (16710308 KB) + Kernel code&data (23834 KB) = CPU addressable DRAM (16734142 KB).
User available (32805040 KB) + Kernel reserved (682484 KB) = Kernel available (33487524 KB) + Kernel code&data (23834 KB) = CPU addressable DRAM (33511358 KB).
Yay! The unusable space is a much smaller, and fixed amount, of slightly over 42 MB. I can handle that.
EDIT3: Looks like I've assumed too much again. My older PC doesn't conform to the above allocations. It seems that all kernel code and data must reside within the "reserved" space. So, the 23834 KB piece is gobbled up and that leaves a bit over 65 MB unaccounted for. It's only 1.5 MB unaccounted on the older PC!
Full 32x32 double iterated grid: Lowest Exponent = 26, Exponent Average = 31.839.
Compared to an average of 30.957 for the single iterated grid scoring, that's not far off average scores being doubled in value. And lowest is half value.
So, based on this one candidate, double iterating destabilises the quality a little ... and distribution likewise?
Looking at max scores, single iterated has one 16G for 15-bit, several 8G for 7-bit, plus 3-, 2- and 1-bit. Double iterated has four 32G for 29-bit and one for 23-bit, with more maximums for 15-bit and under than single-iterated, as would be expected.
Some double-iterated scores are lower, e.g. 8-bit [7:0]. Would it be worthwhile appending +/-/= to the double-iterated scores to show how they compare to single?
As for the distribution, the only one we have is 32-bit [31:0]. As the period is 2^32-1, each non-zero output would occur exactly once and zero never if they were equidistributed, which is not the case and the distribution is a binomial one instead. I don't know what the expected distributions are for less than 32-bit. The average frequency for 16-bit is 2^16, but there is no equation that can predict the distribution that I know of and the results will vary for each lsb, as my tests have shown.
Running full double-iterated grid tests for all non-culled candidates might not be the best use of the resources at the moment. I'd like to know:
1. Does the 32-bit distribution (pair/XORO32) vary with different lsb's for [14,2,7,5]? The ones to try are 5 or 19 or 20 as these have 2G score and 0 is only 256M. The C code would need a new constant to rotate the 32-bit output before incrementing the 4GB byte array.
2. How long does each 32-bit distribution test take? About a minute? I'd prefer to see all the [31:0] pair and zero distributions before any more PractRand grid tests. Which [a,b,c,d] is closest to the ideal? Currently it's [3,2,6,5] for pair frequency but lots of candidates have not been tested yet.
3. What is the distribution for scro's generator with 32-bit output and max score of 32G? Is it a binomial?
Comments
Thanks, Evan. [14,2,7]+p pair/XORO32 distribution is better than [14,2,7]+. Table updated at
http://forums.parallax.com/discussion/comment/1441593/#Comment_1441593
Above results confirm several points:
1. Best xoroshiro32+ triple is [14,2,7].
2. Parity trick makes a huge improvement to bit 0.
3. Bit 1 is much better than bit 0 without parity but weaker than higher bits.
4. Improved bit 0 with parity masks relatively weak bit 1 for [7:0].
PractRand scores, single iteration
Pair/XORO32 frequency, double iteration
Actual and Expected
Pair/XORO32 frequency, double iteration
|Actual-Expected|/Expected
__________________________________________________________________________________________________________ PractRand scoring of Xoroshiro32(16)+ candidate [14 2 7]. Byte Sampled Double Full Period = 8 GB PractRand v0.93 options: stdin -multithreaded -te 1 -tf 2 -tlmin 1KB |===00====01====02====03====04====05====06====07====08====09====10====11====12====13====14====15= 16 | 64K 2M 2M 2M 2M 2M 2M 2M 64K 1M 1M 1M 1M 256K 256K 256K 15 | 256M 256M 256M 256M 256M 256M 256M 256M 256M 256M 512M 512M 512M 512M 512M 512M 14 | 64M 128M 128M 128M 128M 128M 128M 128M 128M 128M 128M 128M 128M 128M 64M 64M 13 | 32M 64M 64M 64M 64M 64M 128M 128M 64M 64M 128M 256M 256M 256M 256M 32M 12 | 2M 8M 16M 16M 64M 2M 2M 2M 2M 2M 2M 2M 2M 2M 2M 2M 11 | 512M 256M 128M 256M 256M 256M 128M 256M 128M 256M 512M 1G 4G 512M 1G 1G 10 | 256M 32M 32M 512M 512M 512M 512M 128M 256M 256M 512M 512M 1G 1G 256M 1G 09 | 1G 1G 128M 128M 2G 2G 4G 4G 2G 2G 2G 2G 2G 2G 1G 2G 08 | 64K 16M 256M 128M 256M 256M 256M 512M 512M 256K 256K 256K 256K 256K 256K 256K 07 | 256M 256M 512M 1G 512M 512M 512M 256M 256M 256M 512M 1G 1G 1G 1G 1G 06 | 256K 32M 64M 128M 128M 128M 256M 256M 64M 128M 128M 256K 256K 256K 256K 256K 05 | 512M 1G 128M 256M 256M 1G 1G 256M 256M 256M 256M 256M 256M 1G 1G 1G 04 | 64K 16M 256M 64M 256M 512M 256M 256M 1G 512M 512M 512M 256M 64K 64K 64K 03 | 256K 128M 1G 1G 1G 1G 1G 2G 1G 1G 512M 2G 1G 1G 256K 256K 02 | 8K 16M 256M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 8K 01 | 8K 2M 64M 128M 512M 1G 1G 1G 1G 1G 256M 512M 1G 512M 1G 1G Lowest Exponent = 13 Exponent Average = 26.335Thanks, Evan. Do you think you might have time this week to do 16-bit double iteration tests for [14,2,7,5]? The results would have practical use on the P2 v1 and FPGA implementions. The three scores needed are [15:0], [31:16] and whichever other one is the highest.
What's the chances of doing any more later? If this is likely to recur I should probably work on getting the automation to do it. EDIT: We could grid them all too then.
[15:0], [31:16] and [25:10] if individual, just to see the difference. Automated grid to test all 16-bit subsamples (and possibly higher) could be handy, though.
Latest scoring logic:
if [ ! -f "$PRreport" ]; then # If no pre-existing Practrand report then run and score the case "$RNGbin" | stdbuf -o L $PRcommand >"$PRreport" if [ $? -ne 0 ]; then printf "Aborted ${RNGbin} - PractRand error\n" exit 2 fi fi # Output the score to console/logfile. This is not score extraction for score table. extract_score if [ $bcfncnt -eq $failcnt ]; then # probably incorrect score due to too sensitive BCFN testing if [ $bcfncnt -gt 2 ];then printf "Passing because " else # rerun testing from +1 power mv "$PRreport" "${PRreport}.tmp" printf "BCFNs: $bcfncnt, ${RNGbin}, PractRand score: ${scoresize}B - Trying larger ...\n" scoresizedn=$scoresize sizeup=$(( $sizekb * 2 )) "$RNGbin" | stdbuf -o L $PRcommand -tlmin "${sizeup}KB" >"$PRreport" extract_score if [ $sizekb -eq $sizeup ] && [ $bcfncnt -gt 0 ]; then # recuring fails if [ $bcfncnt -lt $failcnt ] || [ $bcfncnt -gt 2 ]; then # initial score was valid, revert rm "$PRreport" mv "${PRreport}.tmp" "$PRreport" printf "BCFNs: $bcfncnt, Fails: $failcnt, Reverted - " scoresize=$scoresizedn bcfncnt=0 else # two BCFN glitches in a row! raise another power, rerun again printf "BCFNs: $bcfncnt, Fails: $failcnt, ${RNGbin} - Larger again ...\n" sizeup=$(( $sizekb * 2 )) "$RNGbin" | stdbuf -o L $PRcommand -tlmin "${sizeup}KB" >"$PRreport" extract_score if [ $sizekb -eq $sizeup ]; then # plain giving up at three tries, revert rm "$PRreport" mv "${PRreport}.tmp" "$PRreport" printf "BCFNs: $bcfncnt, Fails: $failcnt, Reverted - " scoresize=$scoresizedn bcfncnt=0 else # got through it, any more glitches for this case won't be detected on this run rm "${PRreport}.tmp" fi fi else # revised report is normal, BCFN glitch cleared rm "${PRreport}.tmp" if [ $bcfncnt -lt $failcnt ]; then # report the correction printf "BCFNs: $bcfncnt, Fails: $failcnt, Accepted - " fi fi fi if [ $bcfncnt -eq $failcnt ]; then printf "BCFN & Fails: $failcnt, " fi fi printf "${RNGbin}, PractRand score: ${scoresize}B\n" rm "$RNGbin"Is is Thursday already? Time flies!
While working my way through the scripts it started getting messy enough that I decided to do a cleaning round ... and that proved troublesome due to ripple effects right throughout.
Had a day off. Started a functioning culling run today - Looking to be twice the number of passing candidates with 512 MB threshold. Again, none of the [14 2 7 x] candidates made the grade.
I've only just finished the gridding clean-up right now. I'll have a grid done for [14 2 7 5] first ...
Full 16x16 single iterated grid: Lowest Exponent = 27, Exponent Average = 30.957
Even 8x16 single iterated grid: Lowest Exponent = 27, Exponent Average = 30.242
Even 16x32 double iterated grid: Lowest Exponent = 26, Exponent Average = 31.126
Note the heightened average even though the worst case is lower. And it'll be higher again with the odds included.
__________________________________________________________________________________________________________ Gridded scores of single iterated Xoroshiro32(16)++ candidate [14 2 7 5]. Byte Sampled Double Full Period = 8 GB PractRand v0.93 options: stdin -multithreaded -te 1 -tf 2 -tlmin 1KB |===00====01====02====03====04====05====06====07====08====09====10====11====12====13====14====15= 16 | 256M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 15 | 4G 2G 8G 8G 4G 4G 4G 2G 4G 4G 4G 8G 4G 8G 4G 16G 14 | 4G 4G 4G 2G 2G 2G 1G 2G 1G 2G 2G 2G 4G 4G 4G 2G 13 | 2G 4G 4G 4G 4G 4G 4G 4G 4G 2G 4G 2G 4G 4G 2G 2G 12 | 512M 512M 1G 1G 512M 512M 1G 1G 1G 512M 1G 1G 1G 1G 512M 512M 11 | 2G 2G 4G 4G 4G 2G 4G 4G 4G 2G 2G 4G 4G 4G 4G 4G 10 | 2G 2G 1G 2G 2G 2G 1G 2G 2G 2G 2G 2G 2G 2G 2G 2G 09 | 2G 4G 4G 4G 4G 8G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 08 | 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 256M 512M 1G 1G 1G 1G 07 | 8G 2G 4G 4G 4G 4G 8G 8G 8G 8G 4G 4G 4G 8G 8G 4G 06 | 4G 2G 512M 2G 1G 2G 2G 4G 2G 2G 2G 2G 1G 2G 4G 2G 05 | 4G 2G 4G 4G 4G 2G 4G 2G 4G 4G 4G 4G 2G 2G 2G 4G 04 | 128M 1G 2G 1G 1G 2G 1G 2G 2G 1G 1G 2G 1G 512M 1G 512M 03 | 2G 4G 2G 4G 4G 4G 2G 4G 4G 4G 4G 4G 4G 4G 4G 2G 02 | 1G 2G 2G 2G 2G 2G 2G 2G 2G 1G 1G 2G 2G 2G 512M 512M 01 | 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G__________________________________________________________________________________________________________ Gridded scores of single iterated Xoroshiro32(16)++ candidate [14 2 7 5]. Byte Sampled Double Full Period = 8 GB PractRand v0.93 options: stdin -multithreaded -te 1 -tf 2 -tlmin 1KB |===00====01====02====03====04====05====06====07====08====09====10====11====12====13====14====15= 16 | 256M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 512M 14 | 4G 4G 4G 2G 2G 2G 1G 2G 1G 2G 2G 2G 4G 4G 4G 2G 12 | 512M 512M 1G 1G 512M 512M 1G 1G 1G 512M 1G 1G 1G 1G 512M 512M 10 | 2G 2G 1G 2G 2G 2G 1G 2G 2G 2G 2G 2G 2G 2G 2G 2G 08 | 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 256M 512M 1G 1G 1G 1G 06 | 4G 2G 512M 2G 1G 2G 2G 4G 2G 2G 2G 2G 1G 2G 4G 2G 04 | 128M 1G 2G 1G 1G 2G 1G 2G 2G 1G 1G 2G 1G 512M 1G 512M 02 | 1G 2G 2G 2G 2G 2G 2G 2G 2G 1G 1G 2G 2G 2G 512M 512M__________________________________________________________________________________________________________ Gridded scores of double iterated Xoroshiro32(16)++ candidate [14 2 7 5]. Byte Sampled Double Full Period = 8 GB PractRand v0.93 options: stdin -multithreaded -te 1 -tf 2 -tlmin 1KB |===00====01====02====03====04====05====06====07====08====09====10====11====12====13====14====15====16====17====18====19====20====21====22====23====24====25====26====27====28====29====30====31= 32 | 256M 1G 1G 1G 1G 2G 512M 512M 512M 256M 256M 256M 1G 1G 1G 1G 256M 1G 1G 2G 2G 1G 512M 512M 1G 1G 512M 1G 1G 1G 1G 1G 30 | 8G 8G 4G 8G 4G 16G 8G 16G 8G 16G 8G 8G 4G 8G 4G 8G 4G 4G 4G 4G 4G 4G 4G 8G 8G 4G 4G 8G 8G 16G 8G 8G 28 | 2G 2G 4G 4G 2G 4G 1G 4G 2G 4G 4G 4G 2G 1G 2G 8G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 4G 4G 4G 26 | 8G 8G 4G 8G 4G 16G 8G 16G 8G 16G 8G 8G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 8G 8G 24 | 2G 2G 2G 2G 2G 2G 2G 4G 2G 2G 2G 2G 2G 2G 2G 4G 2G 2G 2G 2G 2G 4G 4G 2G 2G 2G 2G 2G 2G 2G 4G 2G 22 | 2G 2G 2G 4G 4G 4G 4G 4G 8G 8G 8G 8G 4G 4G 4G 8G 8G 8G 8G 4G 4G 8G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 20 | 2G 1G 1G 1G 2G 4G 2G 2G 2G 4G 2G 2G 1G 2G 2G 2G 2G 4G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 1G 2G 2G 2G 18 | 4G 2G 4G 2G 2G 4G 4G 4G 4G 4G 8G 8G 2G 2G 4G 4G 4G 8G 4G 4G 16G 4G 4G 4G 8G 8G 4G 4G 4G 2G 2G 4G 16 | 512M 1G 1G 1G 1G 1G 1G 1G 512M 256M 128M 128M 1G 1G 512M 512M 512M 512M 512M 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 14 | 2G 2G 2G 4G 2G 16G 2G 4G 2G 4G 8G 16G 4G 4G 2G 2G 2G 2G 2G 4G 2G 2G 4G 8G 4G 8G 8G 16G 16G 8G 4G 4G 12 | 1G 512M 1G 1G 512M 1G 4G 2G 1G 1G 1G 1G 2G 2G 2G 2G 512M 1G 1G 1G 512M 1G 1G 256M 256M 1G 1G 4G 4G 8G 4G 2G 10 | 2G 8G 4G 2G 2G 1G 1G 1G 1G 2G 4G 8G 8G 1G 512M 1G 2G 8G 4G 4G 2G 1G 1G 2G 1G 1G 2G 8G 4G 8G 512M 1G 08 | 64M 512M 1G 2G 512M 1G 128M 128M 1G 2G 4G 4G 4G 2G 2G 256M 64M 512M 1G 2G 512M 1G 128M 128M 1G 4G 2G 1G 1G 2G 2G 256M 06 | 2G 2G 2G 2G 4G 2G 4G 8G 4G 2G 4G 4G 2G 4G 4G 4G 2G 2G 2G 2G 4G 2G 4G 4G 4G 2G 4G 8G 2G 4G 4G 4G 04 | 2G 1G 1G 2G 4G 1G 2G 1G 4G 1G 2G 2G 1G 1G 2G 2G 2G 1G 1G 2G 4G 1G 2G 1G 4G 1G 2G 2G 1G 1G 2G 2G 02 | 2G 1G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 512M 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 512M 2G 2G 2GThe subsamples with the greatest frequency variations have the lowest PractRand scores. This is the opposite of what I predicted they might be.
I gather you've got access to another computer to generate those numbers with.
I guess you could say we've done a pretty good job verifying distribution scoring as a rapid candidate selection method. My Practrand based mass scoring approach has been a good test bed for doing the proving but it isn't practical to carry on using for longer worded engines. Although shorter engines would be fine.
Compared to an average of 30.957 for the single iterated grid scoring, that's not far off average scores being doubled in value. And lowest is half value.
So, based on this one candidate, double iterating destabilises the quality a little ... and distribution likewise?
__________________________________________________________________________________________________________ Gridded scores of double iterated Xoroshiro32(16)++ candidate [14 2 7 5]. Byte Sampled Double Full Period = 8 GB PractRand v0.93 options: stdin -multithreaded -te 1 -tf 2 -tlmin 1KB |===00====01====02====03====04====05====06====07====08====09====10====11====12====13====14====15====16====17====18====19====20====21====22====23====24====25====26====27====28====29====30====31= 32 | 256M 1G 1G 1G 1G 2G 512M 512M 512M 256M 256M 256M 1G 1G 1G 1G 256M 1G 1G 2G 2G 1G 512M 512M 1G 1G 512M 1G 1G 1G 1G 1G 31 | 8G 8G 16G 8G 16G 8G 16G 16G 16G 16G 16G 16G 16G 16G 16G 16G 8G 8G 8G 8G 8G 8G 16G 16G 16G 8G 16G 16G 16G 16G 16G 16G 30 | 8G 8G 4G 8G 4G 16G 8G 16G 8G 16G 8G 8G 4G 8G 4G 8G 4G 4G 4G 4G 4G 4G 4G 8G 8G 4G 4G 8G 8G 16G 8G 8G 29 | 32G 16G 16G 16G 16G 4G 8G 16G 16G 16G 32G 8G 8G 8G 32G 32G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 16G 16G 16G 16G 28 | 2G 2G 4G 4G 2G 4G 1G 4G 2G 4G 4G 4G 2G 1G 2G 8G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 4G 4G 4G 27 | 16G 16G 8G 4G 8G 16G 8G 8G 8G 8G 16G 16G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 16G 16G 26 | 8G 8G 4G 8G 4G 16G 8G 16G 8G 16G 8G 8G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 8G 8G 25 | 8G 16G 8G 8G 8G 8G 8G 8G 8G 8G 16G 16G 8G 16G 16G 16G 16G 16G 16G 8G 16G 8G 8G 8G 8G 8G 8G 8G 8G 8G 16G 16G 24 | 2G 2G 2G 2G 2G 2G 2G 4G 2G 2G 2G 2G 2G 2G 2G 4G 2G 2G 2G 2G 2G 4G 4G 2G 2G 2G 2G 2G 2G 2G 4G 2G 23 | 8G 8G 16G 4G 4G 4G 4G 4G 4G 4G 8G 16G 8G 8G 16G 16G 32G 16G 8G 8G 16G 16G 8G 8G 8G 8G 8G 8G 8G 8G 8G 8G 22 | 2G 2G 2G 4G 4G 4G 4G 4G 8G 8G 8G 8G 4G 4G 4G 8G 8G 8G 8G 4G 4G 8G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 21 | 4G 4G 4G 8G 16G 4G 4G 4G 4G 4G 8G 16G 8G 8G 8G 8G 16G 16G 16G 8G 8G 8G 8G 8G 8G 8G 8G 4G 8G 8G 8G 8G 20 | 2G 1G 1G 1G 2G 4G 2G 2G 2G 4G 2G 2G 1G 2G 2G 2G 2G 4G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 1G 2G 2G 2G 19 | 8G 4G 4G 4G 8G 16G 16G 4G 4G 4G 8G 8G 4G 8G 8G 8G 16G 8G 16G 16G 8G 8G 8G 16G 16G 16G 8G 8G 8G 4G 8G 8G 18 | 4G 2G 4G 2G 2G 4G 4G 4G 4G 4G 8G 8G 2G 2G 4G 4G 4G 8G 4G 4G 16G 4G 4G 4G 8G 8G 4G 4G 4G 2G 2G 4G 17 | 8G 4G 4G 8G 4G 4G 4G 8G 4G 2G 4G 16G 4G 4G 8G 8G 8G 8G 16G 8G 16G 16G 8G 8G 16G 8G 16G 16G 16G 8G 8G 8G 16 | 512M 1G 1G 1G 1G 1G 1G 1G 512M 256M 128M 128M 1G 1G 512M 512M 512M 512M 512M 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 15 | 4G 8G 8G 8G 8G 8G 4G 8G 16G 8G 4G 16G 8G 8G 8G 4G 4G 8G 8G 16G 16G 16G 16G 16G 8G 16G 16G 8G 16G 16G 8G 8G 14 | 2G 2G 2G 4G 2G 16G 2G 4G 2G 4G 8G 16G 4G 4G 2G 2G 2G 2G 2G 4G 2G 2G 4G 8G 4G 8G 8G 16G 16G 8G 4G 4G 13 | 4G 4G 8G 4G 8G 8G 8G 4G 4G 8G 8G 16G 16G 16G 8G 4G 4G 4G 8G 4G 4G 4G 4G 4G 8G 8G 16G 16G 16G 16G 8G 8G 12 | 1G 512M 1G 1G 512M 1G 4G 2G 1G 1G 1G 1G 2G 2G 2G 2G 512M 1G 1G 1G 512M 1G 1G 256M 256M 1G 1G 4G 4G 8G 4G 2G 11 | 4G 4G 4G 4G 4G 2G 4G 4G 8G 4G 8G 8G 8G 2G 4G 8G 4G 4G 4G 4G 4G 2G 8G 2G 2G 4G 8G 8G 8G 8G 8G 8G 10 | 2G 8G 4G 2G 2G 1G 1G 1G 1G 2G 4G 8G 8G 1G 512M 1G 2G 8G 4G 4G 2G 1G 1G 2G 1G 1G 2G 8G 4G 8G 512M 1G 09 | 1G 8G 8G 8G 4G 512M 512M 512M 4G 8G 8G 8G 8G 4G 1G 512M 1G 8G 8G 8G 4G 512M 512M 1G 8G 8G 8G 8G 8G 8G 1G 512M 08 | 64M 512M 1G 2G 512M 1G 128M 128M 1G 2G 4G 4G 4G 2G 2G 256M 64M 512M 1G 2G 512M 1G 128M 128M 1G 4G 2G 1G 1G 2G 2G 256M 07 | 512M 1G 4G 4G 8G 4G 4G 256M 2G 8G 8G 8G 4G 8G 8G 8G 256M 512M 8G 4G 8G 4G 8G 256M 2G 8G 8G 8G 8G 8G 8G 8G 06 | 2G 2G 2G 2G 4G 2G 4G 8G 4G 2G 4G 4G 2G 4G 4G 4G 2G 2G 2G 2G 4G 2G 4G 4G 4G 2G 4G 8G 2G 4G 4G 4G 05 | 4G 4G 4G 4G 4G 4G 4G 4G 4G 2G 4G 4G 4G 4G 4G 2G 4G 4G 4G 4G 4G 4G 4G 4G 4G 2G 4G 4G 4G 4G 4G 4G 04 | 2G 1G 1G 2G 4G 1G 2G 1G 4G 1G 2G 2G 1G 1G 2G 2G 2G 1G 1G 2G 4G 1G 2G 1G 4G 1G 2G 2G 1G 1G 2G 2G 03 | 4G 2G 2G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 4G 2G 2G 4G 4G 4G 4G 4G 2G 4G 4G 4G 4G 4G 4G 2G 02 | 2G 1G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 512M 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 2G 512M 2G 2G 2G 01 | 512M 1G 1G 1G 1G 1G 1G 1G 1G 512M 1G 1G 1G 1G 1G 1G 512M 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1GAnd here's with one DIMM removed. Now the unusable amount is down to about 340 MB. I totally don't get why it's dynamic at all.
Back to random number testing now.
EDIT2: Big Oops! Those Physical RAM sizes weren't actually truly the amount of DRAM addressable. Asking google the right question made a huge difference to my understanding. Turns out those numbers exclude the whole Linux kernel space! And that has a reserved space that sizes as a percentage of the physical RAM installed.
Here's some corrected numbers: User available (16289976 KB) + Kernel reserved (420332 KB) = Kernel available (16710308 KB)
+ Kernel code&data (23834 KB) = CPU addressable DRAM (16734142 KB).
Installed DRAM (16 GB = 16777216 KB) - Addressable (16734142 KB) = 43074 KB unaccounted for.
User available (32805040 KB) + Kernel reserved (682484 KB) = Kernel available (33487524 KB)
+ Kernel code&data (23834 KB) = CPU addressable DRAM (33511358 KB).
Installed DRAM (32 GB = 33554432 KB) - Addressable (33511358 KB) = 43074 KB unaccounted for.
Yay! The unusable space is a much smaller, and fixed amount, of slightly over 42 MB. I can handle that.
EDIT3: Looks like I've assumed too much again. My older PC doesn't conform to the above allocations. It seems that all kernel code and data must reside within the "reserved" space. So, the 23834 KB piece is gobbled up and that leaves a bit over 65 MB unaccounted for. It's only 1.5 MB unaccounted on the older PC!
Evan, I added your PractRand scores. I can test low and high frequencies up to 16 bits.
Looking at max scores, single iterated has one 16G for 15-bit, several 8G for 7-bit, plus 3-, 2- and 1-bit. Double iterated has four 32G for 29-bit and one for 23-bit, with more maximums for 15-bit and under than single-iterated, as would be expected.
Some double-iterated scores are lower, e.g. 8-bit [7:0]. Would it be worthwhile appending +/-/= to the double-iterated scores to show how they compare to single?
As for the distribution, the only one we have is 32-bit [31:0]. As the period is 2^32-1, each non-zero output would occur exactly once and zero never if they were equidistributed, which is not the case and the distribution is a binomial one instead. I don't know what the expected distributions are for less than 32-bit. The average frequency for 16-bit is 2^16, but there is no equation that can predict the distribution that I know of and the results will vary for each lsb, as my tests have shown.
Running full double-iterated grid tests for all non-culled candidates might not be the best use of the resources at the moment. I'd like to know:
1. Does the 32-bit distribution (pair/XORO32) vary with different lsb's for [14,2,7,5]? The ones to try are 5 or 19 or 20 as these have 2G score and 0 is only 256M. The C code would need a new constant to rotate the 32-bit output before incrementing the 4GB byte array.
2. How long does each 32-bit distribution test take? About a minute? I'd prefer to see all the [31:0] pair and zero distributions before any more PractRand grid tests. Which [a,b,c,d] is closest to the ideal? Currently it's [3,2,6,5] for pair frequency but lots of candidates have not been tested yet.
3. What is the distribution for scro's generator with 32-bit output and max score of 32G? Is it a binomial?
EDIT: Or was it based on period length? I never really studied the purpose.
EDIT2: Array index is formed from the pairing: So that means a 32-bit output word would require 64-bit indexes.