The problem would happen if you, say, had a 9 bit report that goes from -128 to +511 or something. That could encode it's minimum as 0x80 and therefore you would mistakenly interpret as +128. As I say, never really happens I think.
So please stop giving me a headache and test if the latest git version works for you.
You cannot have 9bit signed from -128 to 511... that needs 10 bit. Then minimum is $380, %11_1000_0000, maximum is $1FF, %01_1111_1111. we are extending at bit #9
You can have 9 bit signed from -128 to 255, that's the max signed 9 bit, then the minimum is $180, 1_1000_0000, maximum is $0FF, %0_1111_1111, we are extending at bit #8
I have another "problem" : the normalization should be symmetric for digital joys. -32767..32767 and not -32768..32767, orelse the neutral position is -1 instead of 0
@pik33 said:
I have another "problem" : the normalization should be symmetric for digital joys. -32767..32767 and not -32768..32767, orelse the neutral position is -1 instead of 0
The neutral is never zero. The 8 bit sticks zero at -128. You must ignore such small values anyways, because some devices do not have an inherent deadzone (there's also some XInput devices that bake in weird normalization into their 16 bit reports (my Logitech F710 reports some whacky values when in digital mode)).
I needed a simple way to get/peek to the latest key pressed
The problem is: my Basic interpreter tests the keyboard to stop on ctrl-c. It used get_key() for this and it worked until I started to implement input functions.
If runtime loop eats a key at every line, there are eaten out strings on input
The first idea was simple, use peek_key in the run loop. But this caused another problem: if there is no "character eaters" in the program, and the user press anything, that anything will be on the top of the queue forever anc ctrl-c can no more be detected.
One of solutions is of course to build my own queue inside the interpreter, but if I can look at the latest key pressed, then the runtime loop can stop the program on ctrl-c and not interfere with input procedures.
I wrote a quick and dirty workaround, adding this:
PUB peek_latest_key() : r | head
head := keyq_head
if keyq_tail == head
return 0 ' No event
else
return keyq_data[head]
and it works for the Basic, but what happens if the queue is filled up ? Is the answer 'newest key events lost' correct?
I have an older (late 2000s?) Logitech Extreme 3D Pro, which works equally well before or after your detection fix for newer ones. The hidpad_to_vga.spin2 demo only detects and reports values for 3 of the 4 axes (X, Y, and RZ, but not Z), and it correctly detects all 12 buttons but only reports the states of the first 6, with the rest always reading 0.
I can see all buttons of the new Logitech joystick. It also reports x,y,and rz. (not z). The slider is not visible by the driver but it should be reported somewhere.
New version 1.1.0 on github, now supports Slider and Wheel inputs. Demo only shows slider because it's running out of screen space. Also had to (poorly) add parser state push/pop support because logitech just couldn't keep those opcodes in their pants.
It is also good for playing River Raid on an Atari 8-bit emulator: first attempt, 10th bridge reached. A heavy bottom gives the joystick a good stability and it has also a high precision
New driver tested, the slider works. What is the wheel? One of these?
Everything on my old Logitech Extreme 3D Pro now works with v1.1.0! It reports the value of what I had always called the Z axis in your new Sl axis. All 12 buttons apparently do work; the 6 base buttons are just worn out and started working somewhat after lots of repeated pushing.
In reality it has only 2 buttons that are reported at positions 0 and 1. 2 joysticks are x,y (right) ,z and rx (left). 4 switches at the back are ry, rz, s1 (and I think the 4th switch will be a "wheel". That what looks like a hat and these 3 buttons are intended for controlling the controller itself and they are not reported.
Not one that has all the features and fixes usbnew has. You can maybe possibly make it work by removing all the if/end sequences, conforming with your intended configuration. If memory usage is your concern, try flexspin's -2nu dynamic bytecode mode. As of 6.5.1, usbnew certainly works.
@Wuerfel_21 Thanks, that's what I though. Seems my old Megayume has a version of "1CogKbM_rja3b.spin2" that still works with PropTool. It seems to be initialized to send a button report to $40.
Any chance this version also reports joystick position? Looks like the new version has a 16-bit signed value for one of the joysticks, right?
Everything old is bad and sucks (bugs, poorer compatibility, no hub, etc) and is hardcoded for a particular usecase, so you'd have to hack around, anyways. At that point just hack the new driver into working.
The new version can recognize I don't know how many joysticks, (4?) but more than 1. I tested 2 at once, I don't have more. It returns up to 8 axes in range -32768..32767.
It can be also hacked to return touch positions from a Waveshare touch screen, there is a discussion about it somewhere.
It seems to me that just within the realm of HID devices, devices may be unknowable just by querying them. Is this your guys' experience? I can see the need for drivers to step in and unravel the messiness for an operating system. From what I've been reading in this thread, this seems to be the case.
Is my thinking not right about this, so far? I've mentally struggled with USB for 25 years. I haven't been able to get my head around it. I wish their was some master flow chart that laid out all the dialogs that must go on, in order for this stuff to work. Any advice? Thanks in advance, by the way.
@cgracey said:
It seems to me that just within the realm of HID devices, devices may be unknowable just by querying them. Is this your guys' experience? I can see the need for drivers to step in and unravel the messiness for an operating system. From what I've been reading in this thread, this seems to be the case.
There is only one well established standard for HID devices, the BOOT protocol endpoint for keyboard and mouse. This option was introduced to allow USB keyboard to be used from within the PC BIOS setup with a minimal USB support. Today's BIOS may be a full operating system itself so it may not be necessary but, AFAIK, all keyboards and mouse have this endpoint in addition to the report endpoint. This is what is currently used in our driver too.
There are also the XINPUT and PS3 gamepads that are standardized and can be detected with a simple VID/PID lookup. This is also used in the driver.
Everything else rely on the report descriptor. The problem here is that the meaning of each element (button, axis, hat, etc.) is not defined, so button 0 on a gamepad may be the button labeled A but on another is the button B. This needs a way to consistently map the buttons to the consumer program with a VID/PID lookup. This is what Ada did with the usbnew driver.
I've mentally struggled with USB for 25 years. I haven't been able to get my head around it. I wish their was some master flow chart that laid out all the dialogs that must go on, in order for this stuff to work. Any advice?
@cgracey said:
It seems to me that just within the realm of HID devices, devices may be unknowable just by querying them. Is this your guys' experience? I can see the need for drivers to step in and unravel the messiness for an operating system. From what I've been reading in this thread, this seems to be the case.
What do you mean by "unknowable"? The thing with the HID protocol is that it gives you a lot of information on "how" the device is, but not on "what" it is. You may have a device with 12 buttons and you know which 12 bits those buttons report in, but you don't know what those buttons actually are (is this a face button, shoulder button, center button?). They tried to solve for this (and it works for things like extra keyboard keys), but game controllers generally don't contain any useful data in their descriptor (both because I think Windows could never use it, so no one cared to put it there and because even if it was there, the spec sucks and wouldn't be of much use, either). So you always need to have some way to provide a mapping for at least the buttons. (The basic X/Y main movement can be figured out automatically).
For keyboards and mice, there exists this "boot protocol" that you can enable that has a fixed report format, which allows skipping a lot of complexity. We currently use it for keyboard and mice, though for mice there's an incentive to eventually move to parsing the full report because boot protocol can't read the scroll wheel on all mice (strange under-spec that allows this weird discrepancy). For keyboards it's fine though and a lot of boards don't even have a larger report (since the boot protocol report is already at the limit for low-speed devices, 8 bytes. Though it has an unused byte...). You mostly don't get media keys and stuff like that. (Lately even Microsoft has been too lazy to actually add new HID usages... Try pressing Alt+Win+Ctrl+Shift+L on some unsuspecting Win 10/11 machine. They sell a keyboard that has a combined "all modifiers" key. (Instead of just adding a new key number)).
Speaking of, some game controllers don't use the HID protocol at all and use the XBox 360 protocol instead (we misnomer these "XInput devices" because that's what the manuals will call it, but XInput is really the PC application-side API. The newer XBox controllers use a different thing, but PC-specific controllers remain on the 360 protocol because it has better compatibility). These XInput devices operate similar to a HID device in boot protocol mode, having a fixed report format. It also doesn't have the mapping problem (as long as the device is shaped vaguely similar to a normal 4-button, dual-stick gamepad as the Xbox controller is). Some devices are switchable between HID (here often called "DirectInput", once again by the PC-side API) and XInput by some means.
Is my thinking not right about this, so far? I've mentally struggled with USB for 25 years. I haven't been able to get my head around it. I wish their was some master flow chart that laid out all the dialogs that must go on, in order for this stuff to work. Any advice? Thanks in advance, by the way.
I don't really have it down 100% either, but it's something like this (for an interrupt-type device, like HID is)
device pulls up D+ or D- (this indicates either fullspeed or lowspeed)
host notices this either by looking at its own root port(s) or by polling for a hub's status
host enables port (through hub control packet if neccessary)
host sends SetAddress packet to assign the new device an address
host reads descriptors from the device and uses these to decide how to handle the device (the device descriptor contains the all-important Vendor and Product IDs (VID/PID) that are (supposedly) unique to any particular device model)
host sends SetConfiguration packet to initialize the device (having more than one useful configuration is rare - some 4G modems have this so they detect as a storage device first, which contains their own drivers. Kinda clever.)
host may set up other stuff such as SetProtocol (to enable boot protocol), SetIdle (???), etc.
device is now fully set up
host repeatedly sends IN requests to the HID interface (its ID is read from the configuration descriptor) to poll status data from the device. Device may NAK these requests if nothing changed.
Also, while all of this happens the host has to broadcast a SOF packet at ideally exact 1ms intervals. (that is, it shouldn't start a large transaction right before the SOF is due, for that would prevent it from sending the SOF)
Also, any low-speed packet must be preceded by a "preamble", which hubs are looking for to enable low-speed ports (they are normally disabled so the lowspeed devices don't get confused by fullspeed packets). Currently we bitbang this preamble without the smartpin. There may be room for improvement there.
But, another way I used years ago was to snoop on the USB transactions between a PC and device like keyboard.
Then, pretend to be the PC and replay the interactions on power up.
That gets it into a state where it provides reports when prompted that one can decipher...
Very compact driver this way, but probably not as robust...
I recently bought a Raspberry Pi keyboard for my P2 work. I thought it was the ideal keyboard: small, cheap, and with built-in hub.
But it doesn't work with the USBnew driver. Maybe it doesn't support this boot protocol.
Does anyone of the USB connoisseurs know how to make it work?
Thanks for all your input. That really helps me think about this.
I have had a bad attitude about USB for its entire existence, which hasn't been productive. I see a lot of utility getting locked up and disappeared by it. I mean, it's been forever shrouded in mystery and paywalls. I see it primarily as a giant pay-to-play scam that was foisted on computing, to the detrement of small inventors and to the benefit of silicon companies who had budgets to buy into the monopoly and benefit from it. Now, it's the only way to connect something to a computer. No more parallel or serial ports which were simple to operate. I really need a book called "USB for Grouches".
Anyway, if we want to use a modern keyboard or mouse now, we must understand the USB protocol. We've got the basic USB serializer/deserializer in each smart pin pair, but what goes on top of that is a whole other world that I need to learn about.
Does anyone know if there are known ways to spoof the USB-to-serial protocol? I don't know if this is something that has become generic on the PC side, at this point, but FTDI made drivers for Windows, at least, to work with their own chips. Do other USB-to-serial converters have unique drivers, or is there a generic USB device class that Windows and other OS's can leverage these days for generic USB-to-serial operation? I did buy a Beagle USB 12 Proocol Analyzer that traces low- and full-speed activity. I suppose that could be used to figure it out, though I believe FTDI has implemented some protocol things to safeguard against knock-offs. There's no "free" way, it seems.
@cgracey said:
Anyway, if we want to use a modern keyboard or mouse now, we must understand the USB protocol.
Or not. Just use the driver as it exists and mostly works and try to forget about the jank and stank inside, lest it haunts you forever. For usbnew I did a number of optimizations to limit RAM usage to a very small footprint (maybe 12K with all features enabled? Need to check that...), so it should fit into any program easily. This is also why it needs if/end in DAT sections. Because there's so much that's specific to one feature that may not be used. For example, if you don't need game controllers, all the game controller-specific code is not compiled.
Comments
The problem would happen if you, say, had a 9 bit report that goes from -128 to +511 or something. That could encode it's minimum as 0x80 and therefore you would mistakenly interpret as +128. As I say, never really happens I think.
So please stop giving me a headache and test if the latest git version works for you.
You cannot have 9bit signed from -128 to 511... that needs 10 bit. Then minimum is $380, %11_1000_0000, maximum is $1FF, %01_1111_1111. we are extending at bit #9
You can have 9 bit signed from -128 to 255, that's the max signed 9 bit, then the minimum is $180, 1_1000_0000, maximum is $0FF, %0_1111_1111, we are extending at bit #8
I have another "problem" : the normalization should be symmetric for digital joys. -32767..32767 and not -32768..32767, orelse the neutral position is -1 instead of 0
The fresh github version works.
The neutral is never zero. The 8 bit sticks zero at -128. You must ignore such small values anyways, because some devices do not have an inherent deadzone (there's also some XInput devices that bake in weird normalization into their 16 bit reports (my Logitech F710 reports some whacky values when in digital mode)).
I think I will reduce that to 8 bit in the interpreter, that's more than enough to control anything except a chirurgical robot or a nuclear warhead
I needed a simple way to get/peek to the latest key pressed
The problem is: my Basic interpreter tests the keyboard to stop on ctrl-c. It used get_key() for this and it worked until I started to implement input functions.
If runtime loop eats a key at every line, there are eaten out strings on input
The first idea was simple, use peek_key in the run loop. But this caused another problem: if there is no "character eaters" in the program, and the user press anything, that anything will be on the top of the queue forever anc ctrl-c can no more be detected.
One of solutions is of course to build my own queue inside the interpreter, but if I can look at the latest key pressed, then the runtime loop can stop the program on ctrl-c and not interfere with input procedures.
I wrote a quick and dirty workaround, adding this:
PUB peek_latest_key() : r | head head := keyq_head if keyq_tail == head return 0 ' No event else return keyq_data[head]and it works for the Basic, but what happens if the queue is filled up ? Is the answer 'newest key events lost' correct?
Just use the state interface.
That works (and looks better) - I didn't notice this function earlier.
I have an older (late 2000s?) Logitech Extreme 3D Pro, which works equally well before or after your detection fix for newer ones. The
hidpad_to_vga.spin2demo only detects and reports values for 3 of the 4 axes (X, Y, and RZ, but not Z), and it correctly detects all 12 buttons but only reports the states of the first 6, with the rest always reading 0.I can see all buttons of the new Logitech joystick. It also reports x,y,and rz. (not z). The slider is not visible by the driver but it should be reported somewhere.
The base buttons should definitely work. Confirm functioning on other host (PC etc).
Throttle not being exposed is a known issue. Given there appears to be interest in these sticks, I'll fix that.
New version 1.1.0 on github, now supports Slider and Wheel inputs. Demo only shows slider because it's running out of screen space. Also had to (poorly) add parser state push/pop support because logitech just couldn't keep those opcodes in their pants.
It is also good for playing River Raid on an Atari 8-bit emulator: first attempt, 10th bridge reached. A heavy bottom gives the joystick a good stability and it has also a high precision
New driver tested, the slider works. What is the wheel? One of these?
I actually have no idea. Steering wheels usually just use the X axis.
Everything on my old Logitech Extreme 3D Pro now works with
v1.1.0! It reports the value of what I had always called the Z axis in your newSlaxis. All 12 buttons apparently do work; the 6 base buttons are just worn out and started working somewhat after lots of repeated pushing.FRSky Taranis X-Lite S/Pro - $12094F54:
works:
In reality it has only 2 buttons that are reported at positions 0 and 1. 2 joysticks are x,y (right) ,z and rx (left). 4 switches at the back are ry, rz, s1 (and I think the 4th switch will be a "wheel". That what looks like a hat and these 3 buttons are intended for controlling the controller itself and they are not reported.
Is there a version of the USB driver that works with Prop Tool?
Would like to test usb gamepad in Prop Tool..
Not one that has all the features and fixes usbnew has. You can maybe possibly make it work by removing all the if/end sequences, conforming with your intended configuration. If memory usage is your concern, try flexspin's -2nu dynamic bytecode mode. As of 6.5.1, usbnew certainly works.
@Wuerfel_21 Thanks, that's what I though. Seems my old Megayume has a version of "1CogKbM_rja3b.spin2" that still works with PropTool. It seems to be initialized to send a button report to $40.
Any chance this version also reports joystick position? Looks like the new version has a 16-bit signed value for one of the joysticks, right?
Everything old is bad and sucks (bugs, poorer compatibility, no hub, etc) and is hardcoded for a particular usecase, so you'd have to hack around, anyways. At that point just hack the new driver into working.
Remove conditional directives and it should work.
The new version can recognize I don't know how many joysticks, (4?) but more than 1. I tested 2 at once, I don't have more. It returns up to 8 axes in range -32768..32767.
It can be also hacked to return touch positions from a Waveshare touch screen, there is a discussion about it somewhere.
So, I've read through this entire thread to try to get a mental model of low- and full-speed USB device interfacing.
I've also been reading what I can find on the internet about it all, including HID protocols. Been plowing through this, just now:
https://www.usb.org/sites/default/files/documents/hid1_11.pdf
It seems to me that just within the realm of HID devices, devices may be unknowable just by querying them. Is this your guys' experience? I can see the need for drivers to step in and unravel the messiness for an operating system. From what I've been reading in this thread, this seems to be the case.
Is my thinking not right about this, so far? I've mentally struggled with USB for 25 years. I haven't been able to get my head around it. I wish their was some master flow chart that laid out all the dialogs that must go on, in order for this stuff to work. Any advice? Thanks in advance, by the way.
There is only one well established standard for HID devices, the BOOT protocol endpoint for keyboard and mouse. This option was introduced to allow USB keyboard to be used from within the PC BIOS setup with a minimal USB support. Today's BIOS may be a full operating system itself so it may not be necessary but, AFAIK, all keyboards and mouse have this endpoint in addition to the report endpoint. This is what is currently used in our driver too.
There are also the XINPUT and PS3 gamepads that are standardized and can be detected with a simple VID/PID lookup. This is also used in the driver.
Everything else rely on the report descriptor. The problem here is that the meaning of each element (button, axis, hat, etc.) is not defined, so button 0 on a gamepad may be the button labeled A but on another is the button B. This needs a way to consistently map the buttons to the consumer program with a VID/PID lookup. This is what Ada did with the usbnew driver.
This may help a bit:
https://ftdichip.com/wp-content/uploads/2020/08/TN_113_Simplified-Description-of-USB-Device-Enumeration.pdf
What do you mean by "unknowable"? The thing with the HID protocol is that it gives you a lot of information on "how" the device is, but not on "what" it is. You may have a device with 12 buttons and you know which 12 bits those buttons report in, but you don't know what those buttons actually are (is this a face button, shoulder button, center button?). They tried to solve for this (and it works for things like extra keyboard keys), but game controllers generally don't contain any useful data in their descriptor (both because I think Windows could never use it, so no one cared to put it there and because even if it was there, the spec sucks and wouldn't be of much use, either). So you always need to have some way to provide a mapping for at least the buttons. (The basic X/Y main movement can be figured out automatically).
For keyboards and mice, there exists this "boot protocol" that you can enable that has a fixed report format, which allows skipping a lot of complexity. We currently use it for keyboard and mice, though for mice there's an incentive to eventually move to parsing the full report because boot protocol can't read the scroll wheel on all mice (strange under-spec that allows this weird discrepancy). For keyboards it's fine though and a lot of boards don't even have a larger report (since the boot protocol report is already at the limit for low-speed devices, 8 bytes. Though it has an unused byte...). You mostly don't get media keys and stuff like that. (Lately even Microsoft has been too lazy to actually add new HID usages... Try pressing Alt+Win+Ctrl+Shift+L on some unsuspecting Win 10/11 machine. They sell a keyboard that has a combined "all modifiers" key. (Instead of just adding a new key number)).
Speaking of, some game controllers don't use the HID protocol at all and use the XBox 360 protocol instead (we misnomer these "XInput devices" because that's what the manuals will call it, but XInput is really the PC application-side API. The newer XBox controllers use a different thing, but PC-specific controllers remain on the 360 protocol because it has better compatibility). These XInput devices operate similar to a HID device in boot protocol mode, having a fixed report format. It also doesn't have the mapping problem (as long as the device is shaped vaguely similar to a normal 4-button, dual-stick gamepad as the Xbox controller is). Some devices are switchable between HID (here often called "DirectInput", once again by the PC-side API) and XInput by some means.
See https://github.com/Wuerfel21/usbnew/wiki/PADMAP.TXT-Format-and-the-reference-gamepad#the-reference-gamepad for the solution I came up with for button remapping.
I don't really have it down 100% either, but it's something like this (for an interrupt-type device, like HID is)
Also, while all of this happens the host has to broadcast a SOF packet at ideally exact 1ms intervals. (that is, it shouldn't start a large transaction right before the SOF is due, for that would prevent it from sending the SOF)
Also, any low-speed packet must be preceded by a "preamble", which hubs are looking for to enable low-speed ports (they are normally disabled so the lowspeed devices don't get confused by fullspeed packets). Currently we bitbang this preamble without the smartpin. There may be room for improvement there.
Any further questions?
I like the BOOT protocol driver.
But, another way I used years ago was to snoop on the USB transactions between a PC and device like keyboard.
Then, pretend to be the PC and replay the interactions on power up.
That gets it into a state where it provides reports when prompted that one can decipher...
Very compact driver this way, but probably not as robust...
I recently bought a Raspberry Pi keyboard for my P2 work. I thought it was the ideal keyboard: small, cheap, and with built-in hub.
But it doesn't work with the USBnew driver. Maybe it doesn't support this boot protocol.
Does anyone of the USB connoisseurs know how to make it work?
Andy
I think @pik33 had a Pi keyboard like that working. Anyways, you got the driver working with other devices?
Macca, Wuerfel_21, and Rayman,
Thanks for all your input. That really helps me think about this.
I have had a bad attitude about USB for its entire existence, which hasn't been productive. I see a lot of utility getting locked up and disappeared by it. I mean, it's been forever shrouded in mystery and paywalls. I see it primarily as a giant pay-to-play scam that was foisted on computing, to the detrement of small inventors and to the benefit of silicon companies who had budgets to buy into the monopoly and benefit from it. Now, it's the only way to connect something to a computer. No more parallel or serial ports which were simple to operate. I really need a book called "USB for Grouches".
Anyway, if we want to use a modern keyboard or mouse now, we must understand the USB protocol. We've got the basic USB serializer/deserializer in each smart pin pair, but what goes on top of that is a whole other world that I need to learn about.
Does anyone know if there are known ways to spoof the USB-to-serial protocol? I don't know if this is something that has become generic on the PC side, at this point, but FTDI made drivers for Windows, at least, to work with their own chips. Do other USB-to-serial converters have unique drivers, or is there a generic USB device class that Windows and other OS's can leverage these days for generic USB-to-serial operation? I did buy a Beagle USB 12 Proocol Analyzer that traces low- and full-speed activity. I suppose that could be used to figure it out, though I believe FTDI has implemented some protocol things to safeguard against knock-offs. There's no "free" way, it seems.
USB serial, see this thread: https://forums.parallax.com/discussion/175254/usb-device-serial-device
Or not. Just use the driver as it exists and mostly works and try to forget about the jank and stank inside, lest it haunts you forever. For usbnew I did a number of optimizations to limit RAM usage to a very small footprint (maybe 12K with all features enabled? Need to check that...), so it should fit into any program easily. This is also why it needs if/end in DAT sections. Because there's so much that's specific to one feature that may not be used. For example, if you don't need game controllers, all the game controller-specific code is not compiled.