In search of the Perfect Button Press

I'm making a simple version of a Space Invaders game using an 8x8 grid of LED's.

I've done a bit of experimenting so that I get familiar with the SX, including the program below by adapting a program from Gunther Daubach's excellent book. For the purposes of familiarizing my self with the SX, I wired up a hex switch to the highest 4 bits of PORTC. Every time I press the button it loads up a hex number and stores it into the first 8 memory locations of the General Purpose section of the data memory, just the way I want it too. Here's the problem though:

If I press the button repeatedly, it won't register all the button presses.
If I hold the button down for too long and then let go, it registers the letting-go of the button as a button press
The program has a timer in it, when I'd rather have the program doing something more important.

Like I said, this is for a fun little Space Invaders handheld game I'm making, so I want the user to be able to mash on the button and be able to send a stream of Led's up to the invaders. The program does work pretty good, just not perfectly.

Basically I'm looking for the perfect button press method.

One press, one registration of doing so
No delay
No hogging the ISR

Thanks in advance for any and all help.

Include "setup28.inc"
RESET Main

TRIS    equ    $0f
PLP    equ    $0e
WKEN    equ    $0b
WKPND_W    equ    $09

org $08

Counter1 ds 1
Counter2 ds 1
Counter3 ds 1
Time     ds 1
Bounce     ds 3
localTemp0    ds 1


org    $1e
CopyFSR    ds 1
temp     ds 1


org    $000
;----------------------------------------
; ISR
;----------------------------------------

    mov    localTemp0, m
    ;I have to use a copy of the FSR, because the SX automatically resets the FSR
    ;back to whatever it was when the interrupt started
    mov fsr, CopyFSR 
    mov ind, rc     ;put PortC values into whatever FSR is pointing at
    
;The next section is for checking to see if the fsr has reached the 8th
;memory location.  If it has, the program makes the fsr point at 0x10 of data memory, as it was when
;the SX was reset.  This is because there are going to be 8 strips of LED's.
    
    mov temp,CopyFSR    ;move a CopyFSR into Temp
    inc CopyFSR
    and temp,#0x07    ;get rid of all high bits
    inc temp
    sb  temp.3    ;If CopyFSR has reached 7, then it's time to reset the CopyFSR back to 0x10 of data memory
    jmp Exit_ISR
    mov CopyFSR,#0x10    ;reset fsr back to starting point
Exit_ISR

    clr    Bounce
    clr    Bounce + 1
    mov    Bounce + 2, #$7F    ;When I adjusted the literal in this line, it acted on only one button 
                    ;press like it should
:DeBounce
    decsz    Bounce
    jmp    :DeBounce
    decsz    Bounce +1
    jmp    :DeBounce
    decsz    Bounce + 2
    jmp    :DeBounce


;Now back to administrative stuff for ISR
    mode    WKPND_W        ;this section gets it ready for another interrupt
    clr    w
    mov    !rb, w
    mov    m, localTemp0
    reti

org    $100

Main
    mode    TRIS        ;Configure port b for i/o
    mov    !rb, #%11111110

    mode    TRIS        ;Configure port c all input
    mov    !rc, #$FF

    mode    PLP
    mov    !rb, #%11110111

    mode    WKPND_W        ;Reset interrupt flags on portb
    clr    w
    mov    !rb, w
    
    mode    WKEN        ;Enable rb.3 to cause an interrupt
    mov    !rb, #%11110111

    mov    CopyFSR, #$10    ;I want the data to start at $10 in data memory

Loop
    jmp    Loop