Let's assume the next P2 may be able sample a component analog video signal reasonably well at QVGA resolutions. Do people think it would be possible with up to 8 COGs at our disposal to be able to translate from YUV to RGB and figure out some type of optimal colour palette per frame and stream this data out to SD, maybe via a larger intermediate HyperRAM buffer if required, on the fly? I guess the palette selection algorithm required is the trickiest bit, perhaps some type of histogram per frame is needed then conversion of all pixels in the frame to the "closest" palette entry. If QVGA is used with progressive input formats you will have every second skipped scan line to do further work on the input data too. Maybe some real-time video capture might be possible?
I'm sort of guessing the final result might look rather noisy if the palette algorithm and ADC sampling is not the best, but it would make a pretty amazing demo at minimum.
@rogloh - sounds like a great idea. So when are you going to have this ready?
@cgracey - btw the scrolling speed is of course dependent upon content and in the case of the DIR it is not only just listing the directory, it is also scanning the FAT and reporting actual allocated size as well as file size, and then it opens each file and prints the file header too while skipping these options for very large files. But the character draw and scrolling speeds aren't too bad as this test shows:
TAQOZ# LAP VSCROLL LAP .LAP --- 190,824 cycles= 636,080ns @300MHz ok
TAQOZ# LAP $41 VEMIT LAP .LAP --- 3,256 cycles= 10,853ns @300MHz ok
VSCROLL uses LMOVE which uses setq2 and rdlong/wrlongs in blocks for a very fast move while VEMIT ends up passing the font and screen pointer along with colors to the WRCH p2asm routine that takes 2.2us to render the character.
These are the TAQOZ routines for VGA scrolling and drawing fonts with LMOVE and WRCH as assembly routines.
pri VSCROLL
_scrolls C@
IF SCR scrsz + b/l _scrolls C@ W* - DUP b/l + SWAP b/l _scrolls C@ 1- W* 4/ LMOVE THEN
SCR scrsz + b/l - b/l PAPER@ FILL
lsp NEGATE row +!
;
*** DRAW CHARACTERS ***
--- Display the 5x7 VGA character and update position etc
pub VCH ( char -- )
$1C - 7 W* FONT5X7 +
col @ rm => IF NEWLINE THEN
col @ row @ cols W* + SCR +
PAPER@ WRCH ( font' screen' pen.paper -- )
csp col +!
;
While we are at testing speeds, here are the timings for display the normal and full screen frames in memory.
TAQOZ# LAP VIEWN LAP .LAP --- 329,088 cycles= 1,096,960ns @300MHz ok
TAQOZ# LAP VIEWF LAP .LAP --- 904,848 cycles= 3,016,160ns @300MHz ok
So that's 1ms for QVGA and 3ms to display in full-screen (those routines actually output these frames as I test them interactively)
@rogloh - sounds like a great idea. So when are you going to have this ready?
LOL. Not me at the moment! Looking at these colour optimized palette algorithms makes it look pretty tough to achieve. Perhaps an offline process (still run on the P2) could do it. However then you would first need to write 3x as much data to the SD if 24 bit pixel data is used during the initial capture process to keep up with the input sampling rate. This is about 6.9 MB/s = 55Mbps with 240p30 video. Not sure how easy that will be in SPI mode to sustain, perhaps a nice fresh SD card in 4 bit SD mode could do it on the P2.
Here's one paper discussing the Median Cut method to optimize colour palettes if anyone is contemplating attempting a P2 solution to wow everyone! There's hopefully other faster/simpler approaches.
I've now added audio playback and can play this as a separate file simultaneously with watching the video. In TAQOZ when files are opened it uses the starting sector as a handle so that in the case of the video the frame number is actually a sector address and so too with the audio. Since reading a QVGA frame takes about 23ms or so I had to increase the audio buffer size to 4K so it only needs to replenish a buffer every 45ms or so. The video has some resizing artifacts in it but I've left it like that to concentrate on making it work first, and then optimizing it later.
This is the code for audio playback
pub STOP wavsz ~ ;
pub AUDIO $8000 + 6 PIN DUP WYPIN 7 PIN WYPIN WAITCT1 ;
pub PLAYBUF wavblk# ADO I W@ AUDIO 2 +LOOP wavblk# NEGATE wavsz +! ;
--- player task runs in a cog constantly feeding the dac audio out
pub PLAYER
6 PIN 12 DAC 7 PIN 12 DAC
BEGIN
BEGIN wavsz @ UNTIL
CLKHZ samrate @ / SETCT1
BEGIN
wavbuf PLAYBUF 1 wavflg CLR
wavbuf2 PLAYBUF 2 wavflg CLR
wavsz @ 1 <
UNTIL
1 COGSTOP
AGAIN
;
--- read a wav sector into a buffer
pri GETWAV ( buf -- ) wavsect @ SWAP wavblk# SDRDS wavblk# 9>> wavsect +! ;
--- this routine needs to be polled at least every 40ms or so
pub ?AUDIO
1 wavflg SET? NOT IF wavbuf GETWAV 1 wavflg SET THEN
2 wavflg SET? NOT IF wavbuf2 GETWAV 2 wavflg SET THEN
;
pre PLAY ( <name> -- )
GET$
pub PLAY$
" .WAV" OPEN-EXT$
@FILE 0EXIT
0 SD@ " RIFF" @ = 0EXIT
8 SD@ " WAVE" @ = 0EXIT
@FILE wavsect ! wavbuf GETWAV 1 wavflg C!
wavbuf 44 ERASE --- mute header data
24 SD@ samrate !
40 SD@ wavsz !
1 NEWCOG ' PLAYER 1 RUN
' ?AUDIO keypoll W!
;
While I was creating an audio waveform display which I haven't finished yet since it needs some nice rounded lines to join the dots yet, I nonetheless indulged in playing a bit more and added a simple one line effect that modulates the picture to the music. See what you think.
pub MODULATE POLL: ao W@ 16<< 22 SAR 120 + vint C! ;
Man, those visual effects did a good re-flashing at my own memories.
During six years (68 thru 73) I've used to make some money, running a rental service, with a pair of rhythmic lights effects boxes, two tube-based audio amps and many other light&sound equipment.
Every weekend, two dancing events. Used to be sure hits at the neighbourhood.
While I was creating an audio waveform display which I haven't finished yet since it needs some nice rounded lines to join the dots yet, I nonetheless indulged in playing a bit more and added a simple one line effect that modulates the picture to the music. See what you think.
...
pub MODULATE POLL: ao W@ 16<< 22 SAR 120 + vint C! ;
looks you're really having fun
now next a window-system ...
reminds me of the first SMALLTALK system I saw (and still have somewhere)
for IBM PC with 512 kB RAM on 2 floppies 5-1/4 single sided some 250kB each
... One for the SMALLTALK-IMAGE and one for the SOURCES.
Interactive - with full integrated development system, and 'windows'.
I think it was 1986 ... PC-Magazine 8-1985
Watching that fun stuff Peter's already doing makes me think that perhaps another way to promote the P2 would be for Parallax to organize a proper (i.e. real prize winning) competition for all demoscene type coders at some point in time whenever sufficient P2's are available to more people at a reasonable price, maybe using some much simpler/cheaper board than the P2-ES board so more young people could enter on a whim too. Ideally get Hackaday involved somehow with judging etc. Imagine the amazing demos they could all produce if their attention could only be focussed on it (the hardest part?). Surely a marketing opportunity for the P2, @"Ken Gracey" .
I guess timing it right is the trick. Hold it prematurely, then it may not make a sufficient impact without enough people having enough time to learn P2 to be able to enter the competition or access working tools, complete documentation etc. Yet if you wait too long you are only missing the sales opportunities that may come from it, but it's still better late than never. Probably no rush yet but hopefully within 12 months would be good timing. This whole "Amiga of microcontroller's" line already mentioned by others might be a good angle to push to catch some old school demoscene people's attention too.
Comments
So when's the video capture to SD card coming?
Let's assume the next P2 may be able sample a component analog video signal reasonably well at QVGA resolutions. Do people think it would be possible with up to 8 COGs at our disposal to be able to translate from YUV to RGB and figure out some type of optimal colour palette per frame and stream this data out to SD, maybe via a larger intermediate HyperRAM buffer if required, on the fly? I guess the palette selection algorithm required is the trickiest bit, perhaps some type of histogram per frame is needed then conversion of all pixels in the frame to the "closest" palette entry. If QVGA is used with progressive input formats you will have every second skipped scan line to do further work on the input data too. Maybe some real-time video capture might be possible?
I'm sort of guessing the final result might look rather noisy if the palette algorithm and ADC sampling is not the best, but it would make a pretty amazing demo at minimum.
@cgracey - btw the scrolling speed is of course dependent upon content and in the case of the DIR it is not only just listing the directory, it is also scanning the FAT and reporting actual allocated size as well as file size, and then it opens each file and prints the file header too while skipping these options for very large files. But the character draw and scrolling speeds aren't too bad as this test shows: VSCROLL uses LMOVE which uses setq2 and rdlong/wrlongs in blocks for a very fast move while VEMIT ends up passing the font and screen pointer along with colors to the WRCH p2asm routine that takes 2.2us to render the character.
These are the TAQOZ routines for VGA scrolling and drawing fonts with LMOVE and WRCH as assembly routines.
While we are at testing speeds, here are the timings for display the normal and full screen frames in memory. So that's 1ms for QVGA and 3ms to display in full-screen (those routines actually output these frames as I test them interactively)
LOL. Not me at the moment! Looking at these colour optimized palette algorithms makes it look pretty tough to achieve. Perhaps an offline process (still run on the P2) could do it. However then you would first need to write 3x as much data to the SD if 24 bit pixel data is used during the initial capture process to keep up with the input sampling rate. This is about 6.9 MB/s = 55Mbps with 240p30 video. Not sure how easy that will be in SPI mode to sustain, perhaps a nice fresh SD card in 4 bit SD mode could do it on the P2.
Here's one paper discussing the Median Cut method to optimize colour palettes if anyone is contemplating attempting a P2 solution to wow everyone! There's hopefully other faster/simpler approaches.
https://pdfs.semanticscholar.org/55ae/0b4eba70cb78546b20767d34fb6acd01d222.pdf
This is the code for audio playback
During six years (68 thru 73) I've used to make some money, running a rental service, with a pair of rhythmic lights effects boxes, two tube-based audio amps and many other light&sound equipment.
Every weekend, two dancing events. Used to be sure hits at the neighbourhood.
Thank you very much.
now next a window-system ...
reminds me of the first SMALLTALK system I saw (and still have somewhere)
for IBM PC with 512 kB RAM on 2 floppies 5-1/4 single sided some 250kB each
... One for the SMALLTALK-IMAGE and one for the SOURCES.
Interactive - with full integrated development system, and 'windows'.
I think it was 1986 ...
PC-Magazine 8-1985
so should be doable on the P2 as well.
I guess timing it right is the trick. Hold it prematurely, then it may not make a sufficient impact without enough people having enough time to learn P2 to be able to enter the competition or access working tools, complete documentation etc. Yet if you wait too long you are only missing the sales opportunities that may come from it, but it's still better late than never. Probably no rush yet but hopefully within 12 months would be good timing. This whole "Amiga of microcontroller's" line already mentioned by others might be a good angle to push to catch some old school demoscene people's attention too.