Hello all,
Thanks to @marsmaster for the additional debugging.
Just like @bebement , we are really expecting some clear feedback from Parallax, at least a confirmation / test results.
We also believe there is more than one problem, both at mechanical, electronic, and firmware levels.
As mentioned before, we have 14 Arlo robots, ordered in 3 different batches, assembled by 3 different persons, and all with the same problems from day 1.
@VonSzarvas I have not found anyone with a recent Arlo robot that works.
The sum of all our workarounds is helping, but not enough to actually use the robots for anything serious (still too unreliable).
After having spent so much time debugging on our side, we cannot afford to invest more time on that platform until there is a clear path forward by Parallax.
Best regards
@Alkarex and all others following this thread with similar issues:
To join Publison's call for action, I've also just alerted Parallax customer support to this thread. Again!!!
I'd like to think you'd have seen these issues resolved long ago. This is a terrible situation.
I actually thought there was some development or solution in progress, but it seems that's not been communicated to you. (Or us Forum moderators for that matter!).
I'll make sure the engineer involved get's your email address that is registered in these forums, so he can contact you directly if there are questions outstanding that are preventing a solution being published.
Maybe this issue didn't get into the customer support workflow, although I can't understand why. For sure there has been an unacceptable communication breakdown; we will get this fixed. Please give Publison and me 48 hours.
To update everyone on progress, Parallax has shared an update with the forum moderators that should resolve the issues reported in this thread.
We have agreed with Parallax to do some real-world testing before sharing the details here, and subject to the speed of UPS deliveries, we intend to complete that this week.
Will keep you posted, and will grab some video or images of the "before-and-after" robot behavior, to demonstrate the solution as successful. Publison and I are working together, and checking things on different versions/ages of kit, to make sure you get going asap. Please feel free to PM either of us, or post questions here, in the meantime. We will post more updates soon.
Note: If anyone has contact with Parallax customer support directly, they will of course also be sharing details once they complete internal testing. Our efforts are in addition, as we want to accelerate the process for you guys here.
A little update.... we didn't receive our test gear until Monday (courier / customs delays), and as such the testing will only commence on this Thursday due to other commitments.
So sorry for the delay, but I wanted to keep you up-to-date and assure you we are moving forward.
We were both working on it yesterday, pretty much all day. And I've just arrived to the office to carry on with it today.
The kit I got did have some mechanical challenges which were overcome by enlarging the bolt hole sizes very slightly and carefully aligning the motor when installing to the block. I'm sure the hole-enlarging wasn't essential, but it makes the assembly experience better in my opinion, as the bolts are free moving (not binding/rubbing against the aluminium block), and so easier to get the entire assembly straight.
We want to document a simple way to get the mounting correct, as that is the root of all the issues we believe. That's today's task.
In terms of data, when the blocks were assembled without care of alignment, my Motor1 wheel pulled 1.2A compared with Motor2 at 0.5A for the same movement.
It's entirely possible some users here are pulling even more current and that's what's tripping the DHB-10, especially at low drive speeds, because the mechanical resistance is too high to get the motor started, or as the motor turns it free-spins at certain points of the revolution, and binds at other points, depending on the alignment. (The binding at only certain points of the wheel revolution also explains why some faults occur after a few turns, not immediately)
So... Motor1 for me was poorly aligned and had significant resistance- caused I'm sure because where the shaft meets the motor (that box-cup end on the shaft), there is hardly any tolerance and any slightly mis-alignment of the motor-to-block will have that shaft/motor point under strain. Removing the top bolt and ensuring the bottom 2 bolts were only finger-tight got Motor1 working properly, and pulling about the same current at Motor2.
I didn't get to electrical testing yet, but I can say after about 8 hours of testing the DHB-10 didn't reset or fail once. @Publison and me are now both of the opinion the root of the issue is a mechanical one; be that installation error, part tolerances, or a combination of both (more likely).
That was great, as the same fault occurs on my test hardware within 1 minute each time.
Then I installed the fix to the DHB-10 and after 8 mins the problem has not come back.
I need to check some more things (after an important diversion to watch the Formula 1!), especially using unmodified wheel blocks, but it seems what we've got a solution today!
I pretty much did the mechanical stuff that @VonSzarvas did, but instead of drilling the block, I took the heads of the three bolts to the belt sander to alleviate the interference with the wall of the pocket. I also backed off the bolts about two turns after bottoming out. I used the "Check Encoder" program to run for 60 seconds to check the ticks, file attached). I ran the two motors forward and backwards for four test. The ticks were all within acceptable values.
I also ran @zhowans program for 2 hours with no errors. @VonSzarvas had some current meters connected with his tests. I will do the same tomorrow, but I can only do one at a time.
Back again, and after almost 4 hours running the zhowan code, there are no faults.
Having re-read this thread, it seems our results match what you've all reported, and in short summary we found 2 issues:
Issue 1 is mechanical - if the wheel blocks are not assembled to the right alignment, then the motor will pull significantly higher current, leading to electrical noise and faults.
Issue 2 is electrical - the encoder inputs are too sensitive to the motor noise, with the issue made worse by issue 1.
The issue will only come for certain users in the right conditions, and I'm grateful for zhowan's "stress test" as that made it easy to replicate the problem on the new hardware.
Both issues are easily solved, but I don't know how that will work for helping customers. For today, we shared our results back to Parallax customer support and we asked them to make an announcement for customers. They were eager to get our feedback, so I am sure a plan is in the wings.
@VonSzarvas, @Publison,
Thank you for your efforts!
I have a couple of questions.
1) Were all of your tests performed with the Arlo unloaded on blocks, or was some of the testing performed with the robot driving on the floor with some sort of load?
2) Did Parallax provide you with any newly engineered parts and or software for your testing, or were you testing standard Arlo complete kits which you modified?
Have you tried to move the robot on a rough floor or drive the robot against a wall that prevent it from moving. We observed that the Arlo is very likely to crash in those situation while it is about the change the speed.
1st video has load around 4 kg (the battery pack)
2nd video has added 6 kg load (2 more battery packs, at 5.3kg and 680g)
I didn't stop Arlo between videos, so it ran for about 4 or 5 minutes in all without crashing. On the bench without load it would crash consistenly after 20-30 seconds otherwise!
2) Parallax sent current stock DHB-10 (unmodified), and the instructions to modify it. I wanted to do the mod myself, so I could test the before and after, as I couldn't get my older DHB-10 to crash.
About the hardware, that was our idea with @Publison. I don't know what Parallax were working on themselves in this regard, and they didn't send new blocks (as we both had them already, and I modified what I had).
To try and explain the thinking.... We both had issues screwing in the bolts, and it seemed to us difficult to properly gauge the right tightening torque when the bolt was so hard to screw into the hole (specifically, the bolt head into the pocket area) to start with. After enlarging the hole and pocket by 0.5mm each, the bolts then freely drop into position and were super easy to tighten appropriately.
What I did was hand-tighten them just until the spacer between the motor and block no longer wobbled. Then gave a very slight (perhaps 1/8) turn extra. Looking at the current meter, that seemed about right before the motor started pulling excessive extra current.
@Publison was ahead of me in this, as he'd already figured that grinding down the bolt heads also worked a treat. Basically anything to allow the bolts to insert and turn freely is the aim. I'm sure the motor specification changed slightly at some point, because the difference is so small. Probably a different manufacturer, and so the bolt pattern and current ratings are perhaps a bit different. I'd guess that there was a switch from imperial to metric at some point
Have you tried to move the robot on a rough floor or drive the robot against a wall that prevent it from moving. We observed that the Arlo is very likely to crash in those situation while it is about the change the speed.
I didn't do that. I did physically grab the wheel whilst Arlo was on blocks, and it didn't crash for a short grab. Not terribly scientific I know!
Good point though, that rough floor and incline testing would have tested with different stresses.
That said.... I think Arlo must stop if it hits a wall and cannot go anywhere. That could be dangerous if it didn't? I guess the software is written to stop first, and ask later. Something like a ping sensor should handle the wall avoidance, or in the users software the robot should back off if it encounters resistance..... don't know if that's doable with the API as-is though. Hmm. maybe, if the dist stops going up (or down), then it's time to reduce speed and back off a little, and rotate perhaps. Probably that's what your doing? I'm just learning the possibilities !
I bought my ARLO second hand. It was assembled about 3 years ago. While Parallax did send me a new DHB-10, I did all the initial testing with the original board. I did swap the board in the end to see if there was any difference. There was not.
I ran the tests with the wheels off so I could get to the bolts and play with the torque. I guess the next set of tests would be to run it on the floor. I would have to mount a laptop on it to get access the the terminal to see error codes.
I'm not sure whether @zhowan meant to drive the robot into a wall such that it actually contacts the wall, or such that some anti-collision software causes the motors to rapidly stop or reverse. Such changes of direction are most likely to cause voltage/current transients that could lead to problems. These sorts of scenarios are where I first observed the problems on my robot. I think most of my early problems were also on rugs or carpet, not smooth wood floors. I think a rug/carpet test is necessary as well.
I am also wondering what the exact make and model of the motor sub-assemblies is. They look like they might be automotive window motors to me. It might be useful to get the actual specifications to do some engineering analysis of the motor requirements versus the DHB10 drive transistors. The 1.5amp current draw cited as unusual earlier in the thread does not seem all that high to me. Some of the small robotics motors I looked at are specified to draw 1.5 amps free running.
I've added a video at the previous link with that same zhowan test code running on cobble stones.
We don't have carpets or rugs here, but I tried in the grass and undulating mud, and that seemed OK too, except the test code could do with going faster and for longer periods between each direction change I think, for a better test. At least a random mixture of longer and shorter, faster and slower, movements.
I really don't have time to work on the code through this week, but would be happy to upload and video any test code shared here.
About the motors- As you mention it, I'm certain I read somewhere that these are window motors.
The 1.5A current draw mentioned before was only unusual in that if was 3x higher than the other channel whilst Arlo was on blocks. So I didn't mean that the DHB-10 couldn't drive that sort of load, just that it was an indicator of an alignment issue.
Previously I grabbed the wheels on a gospd(32,32) command, to watch the current rise well beyond 1.5A. I'm not sure that's on video though, so will repeat that test if I get a moment in.
Ha. I didn't enjoy the burned hands after that test last time, so I'd better get some good gloves found too!
Is there an official updated firmware for the DHB-10? I have a few that I want to update if so.
There as only one update by David Carrier in the thread, I hate that I can't quote post numbers anymore. Anyway, I don't think is official as it was only to try some troubleshooting.
I would like to see the firmware released on the Parallax Inc GitHub page to facilitate open source development.
Does anyone have any idea how to request that this happen?
I would like to be able to fork from an "official" GitHub repository and add David Carrier's debug changes, as well as experiment with changes of my own that could potentially be released back into the community.
One idea I have is to add some sort of reset command.
I realize I could just copy the firmware from the website into my own GitHub repository, but I am hesitant to do this as I am not sure what the correct way to do this without violating any licenses or etiquette would be.
I would like to see the firmware released on the Parallax Inc GitHub page to facilitate open source development.
Does anyone have any idea how to request that this happen?
I would like to be able to fork from an "official" GitHub repository and add David Carrier's debug changes, as well as experiment with changes of my own that could potentially be released back into the community.
One idea I have is to add some sort of reset command.
I realize I could just copy the firmware from the website into my own GitHub repository, but I am hesitant to do this as I am not sure what the correct way to do this without violating any licenses or etiquette would be.
I have sent an email to Parallax support and David requesting licensing info on the firmware. Normally it's MIT from most software coming out of Parallax, but the DHB-10 firmware is missing it. The schematic and board layout are both Open Source, so I suspect the firmware is too, but I can't say for sure, not being an employee.
Comments
Thanks to @marsmaster for the additional debugging.
Just like @bebement , we are really expecting some clear feedback from Parallax, at least a confirmation / test results.
We also believe there is more than one problem, both at mechanical, electronic, and firmware levels.
As mentioned before, we have 14 Arlo robots, ordered in 3 different batches, assembled by 3 different persons, and all with the same problems from day 1.
@VonSzarvas I have not found anyone with a recent Arlo robot that works.
The sum of all our workarounds is helping, but not enough to actually use the robots for anything serious (still too unreliable).
After having spent so much time debugging on our side, we cannot afford to invest more time on that platform until there is a clear path forward by Parallax.
Best regards
To join Publison's call for action, I've also just alerted Parallax customer support to this thread. Again!!!
I'd like to think you'd have seen these issues resolved long ago. This is a terrible situation.
I actually thought there was some development or solution in progress, but it seems that's not been communicated to you. (Or us Forum moderators for that matter!).
I'll make sure the engineer involved get's your email address that is registered in these forums, so he can contact you directly if there are questions outstanding that are preventing a solution being published.
Maybe this issue didn't get into the customer support workflow, although I can't understand why. For sure there has been an unacceptable communication breakdown; we will get this fixed. Please give Publison and me 48 hours.
We have agreed with Parallax to do some real-world testing before sharing the details here, and subject to the speed of UPS deliveries, we intend to complete that this week.
Will keep you posted, and will grab some video or images of the "before-and-after" robot behavior, to demonstrate the solution as successful. Publison and I are working together, and checking things on different versions/ages of kit, to make sure you get going asap. Please feel free to PM either of us, or post questions here, in the meantime. We will post more updates soon.
Note: If anyone has contact with Parallax customer support directly, they will of course also be sharing details once they complete internal testing. Our efforts are in addition, as we want to accelerate the process for you guys here.
We will be testing over the weekend. Thank You for your patience.
A little update.... we didn't receive our test gear until Monday (courier / customs delays), and as such the testing will only commence on this Thursday due to other commitments.
So sorry for the delay, but I wanted to keep you up-to-date and assure you we are moving forward.
We will post news this week!
Any update?
Thanks,
Brad
We were both working on it yesterday, pretty much all day. And I've just arrived to the office to carry on with it today.
The kit I got did have some mechanical challenges which were overcome by enlarging the bolt hole sizes very slightly and carefully aligning the motor when installing to the block. I'm sure the hole-enlarging wasn't essential, but it makes the assembly experience better in my opinion, as the bolts are free moving (not binding/rubbing against the aluminium block), and so easier to get the entire assembly straight.
We want to document a simple way to get the mounting correct, as that is the root of all the issues we believe. That's today's task.
In terms of data, when the blocks were assembled without care of alignment, my Motor1 wheel pulled 1.2A compared with Motor2 at 0.5A for the same movement.
It's entirely possible some users here are pulling even more current and that's what's tripping the DHB-10, especially at low drive speeds, because the mechanical resistance is too high to get the motor started, or as the motor turns it free-spins at certain points of the revolution, and binds at other points, depending on the alignment. (The binding at only certain points of the wheel revolution also explains why some faults occur after a few turns, not immediately)
So... Motor1 for me was poorly aligned and had significant resistance- caused I'm sure because where the shaft meets the motor (that box-cup end on the shaft), there is hardly any tolerance and any slightly mis-alignment of the motor-to-block will have that shaft/motor point under strain. Removing the top bolt and ensuring the bottom 2 bolts were only finger-tight got Motor1 working properly, and pulling about the same current at Motor2.
I didn't get to electrical testing yet, but I can say after about 8 hours of testing the DHB-10 didn't reset or fail once. @Publison and me are now both of the opinion the root of the issue is a mechanical one; be that installation error, part tolerances, or a combination of both (more likely).
So we continue...!
http://forums.parallax.com/discussion/167944/arlobot-dhb-10-motor-board-bug-with-e@zhowan
That was great, as the same fault occurs on my test hardware within 1 minute each time.
Then I installed the fix to the DHB-10 and after 8 mins the problem has not come back.
I need to check some more things (after an important diversion to watch the Formula 1!), especially using unmodified wheel blocks, but it seems what we've got a solution today!
I also ran @zhowans program for 2 hours with no errors. @VonSzarvas had some current meters connected with his tests. I will do the same tomorrow, but I can only do one at a time.
Having re-read this thread, it seems our results match what you've all reported, and in short summary we found 2 issues:
Issue 1 is mechanical - if the wheel blocks are not assembled to the right alignment, then the motor will pull significantly higher current, leading to electrical noise and faults.
Issue 2 is electrical - the encoder inputs are too sensitive to the motor noise, with the issue made worse by issue 1.
The issue will only come for certain users in the right conditions, and I'm grateful for zhowan's "stress test" as that made it easy to replicate the problem on the new hardware.
Both issues are easily solved, but I don't know how that will work for helping customers. For today, we shared our results back to Parallax customer support and we asked them to make an announcement for customers. They were eager to get our feedback, so I am sure a plan is in the wings.
Thank you for your efforts!
I have a couple of questions.
1) Were all of your tests performed with the Arlo unloaded on blocks, or was some of the testing performed with the robot driving on the floor with some sort of load?
2) Did Parallax provide you with any newly engineered parts and or software for your testing, or were you testing standard Arlo complete kits which you modified?
Thanks,
Brad
Thanks for your test,
One more question about the test.
Have you tried to move the robot on a rough floor or drive the robot against a wall that prevent it from moving. We observed that the Arlo is very likely to crash in those situation while it is about the change the speed.
Hi!
1) Mixture. I've uploaded a couple videos here showing the zhowan test running on floor.
https://drive.google.com/drive/folders/1-gS7Gi0WwrkBsP6C6cpaGvELSRUpiFAS
1st video has load around 4 kg (the battery pack)
2nd video has added 6 kg load (2 more battery packs, at 5.3kg and 680g)
I didn't stop Arlo between videos, so it ran for about 4 or 5 minutes in all without crashing. On the bench without load it would crash consistenly after 20-30 seconds otherwise!
2) Parallax sent current stock DHB-10 (unmodified), and the instructions to modify it. I wanted to do the mod myself, so I could test the before and after, as I couldn't get my older DHB-10 to crash.
About the hardware, that was our idea with @Publison. I don't know what Parallax were working on themselves in this regard, and they didn't send new blocks (as we both had them already, and I modified what I had).
To try and explain the thinking.... We both had issues screwing in the bolts, and it seemed to us difficult to properly gauge the right tightening torque when the bolt was so hard to screw into the hole (specifically, the bolt head into the pocket area) to start with. After enlarging the hole and pocket by 0.5mm each, the bolts then freely drop into position and were super easy to tighten appropriately.
What I did was hand-tighten them just until the spacer between the motor and block no longer wobbled. Then gave a very slight (perhaps 1/8) turn extra. Looking at the current meter, that seemed about right before the motor started pulling excessive extra current.
@Publison was ahead of me in this, as he'd already figured that grinding down the bolt heads also worked a treat. Basically anything to allow the bolts to insert and turn freely is the aim. I'm sure the motor specification changed slightly at some point, because the difference is so small. Probably a different manufacturer, and so the bolt pattern and current ratings are perhaps a bit different. I'd guess that there was a switch from imperial to metric at some point
I didn't do that. I did physically grab the wheel whilst Arlo was on blocks, and it didn't crash for a short grab. Not terribly scientific I know!
Good point though, that rough floor and incline testing would have tested with different stresses.
That said.... I think Arlo must stop if it hits a wall and cannot go anywhere. That could be dangerous if it didn't? I guess the software is written to stop first, and ask later. Something like a ping sensor should handle the wall avoidance, or in the users software the robot should back off if it encounters resistance..... don't know if that's doable with the API as-is though. Hmm. maybe, if the dist stops going up (or down), then it's time to reduce speed and back off a little, and rotate perhaps. Probably that's what your doing? I'm just learning the possibilities !
I ran the tests with the wheels off so I could get to the bolts and play with the torque. I guess the next set of tests would be to run it on the floor. I would have to mount a laptop on it to get access the the terminal to see error codes.
I am also wondering what the exact make and model of the motor sub-assemblies is. They look like they might be automotive window motors to me. It might be useful to get the actual specifications to do some engineering analysis of the motor requirements versus the DHB10 drive transistors. The 1.5amp current draw cited as unusual earlier in the thread does not seem all that high to me. Some of the small robotics motors I looked at are specified to draw 1.5 amps free running.
We don't have carpets or rugs here, but I tried in the grass and undulating mud, and that seemed OK too, except the test code could do with going faster and for longer periods between each direction change I think, for a better test. At least a random mixture of longer and shorter, faster and slower, movements.
I really don't have time to work on the code through this week, but would be happy to upload and video any test code shared here.
About the motors- As you mention it, I'm certain I read somewhere that these are window motors.
The 1.5A current draw mentioned before was only unusual in that if was 3x higher than the other channel whilst Arlo was on blocks. So I didn't mean that the DHB-10 couldn't drive that sort of load, just that it was an indicator of an alignment issue.
Previously I grabbed the wheels on a gospd(32,32) command, to watch the current rise well beyond 1.5A. I'm not sure that's on video though, so will repeat that test if I get a moment in.
Ha. I didn't enjoy the burned hands after that test last time, so I'd better get some good gloves found too!
We're going to fix this problem. We're going to replace the DHB-10 boards and likely motor blocks for those having trouble.
I'm still getting up to speed on the solution to the problem, but please know we'll have this resolved before the weekend.
I'll post instructions here on how to go about this. Our team, and VonSzarvas, understand the issue entirely and simply need to address it properly.
You'll hear from me with details between today and tomorrow. Thanks for your patience with this one!
Ken Gracey
Official Firmware is still 1.0 on the website.
https://www.parallax.com/downloads/dhb-10-motor-controller-firmware-guide
Does anyone have any idea how to request that this happen?
I would like to be able to fork from an "official" GitHub repository and add David Carrier's debug changes, as well as experiment with changes of my own that could potentially be released back into the community.
One idea I have is to add some sort of reset command.
I realize I could just copy the firmware from the website into my own GitHub repository, but I am hesitant to do this as I am not sure what the correct way to do this without violating any licenses or etiquette would be.
I have sent an email to Parallax support and David requesting licensing info on the firmware. Normally it's MIT from most software coming out of Parallax, but the DHB-10 firmware is missing it. The schematic and board layout are both Open Source, so I suspect the firmware is too, but I can't say for sure, not being an employee.
If you are having an issue, could you please e-mail kgracey@parallax.com the following:
Number of DHB-10s you have
Complete shipping information for replacement
We will send you the replacement for no charge.
Thanks,
Ken Gracey
Can I just get the latest firmware to update my boards? Or is there a hardware revision?