I was thinking and sometimes that is dangerous. It is about your communication method.
Your P2 sends commands to the legs to move. The legs move sending results.. I think this is too complicated. You should turn this upside down.
Think alike the mailbox system used in spin to cog communication.
Decoupling for parallel execution not serial. take the comms out of the way of thinking.
Say you have a P2 Hub array with all needed parameters and return values for each leg.
On each leg P1 you also have a HUB array with parameter and return values.
Your serial comms from P2 to P1es just sends each leg its Params and receives current return values writing back into P2 HUB. rinse repeat.
On P1es receive new params, write to own Hub read own Hub write current return values. rinse repeat.
This is as simple as it gets. Send X Hub longs to leg, receive X longs from leg, repeat. constantly. even if nothing needs to move.
On the P2 you need no command encoding, sending, reading ... just read you own HUB variables and write your own Hub variables
On the P1 you need no command encoding, sending, reading ... just read you own HUB variables and write your own Hub variables
And then let each leg P1 figure out the movement, give some start and target sysclocks for the whole movement in the HUB array or such.
On the P2 you now can calculate a movement array with proper spacing and - just write into his own Hub, the legs will go there.
think of it as common shared Hub memory between each P1 and the P2.
Sure you have some Latency, but that latency is constant since your serial comms to and from leg runs all the time the same code without stopping or changing.
We command each joint (motor) to be at a certain position at a certain time and voila, we have synchronisation, courtesy of the P1s' clocks.
P2 doesn't need to constantly monitor the positions of each joint. Whatever it last commanded, should be the actual position. If not, the P1 would show "excess position error" by way of the status byte.
We command each joint (motor) to be at a certain position at a certain time and voila, we have synchronisation, courtesy of the P1s' clocks.
P2 doesn't need to constantly monitor the positions of each joint. Whatever it last commanded, should be the actual position. If not, the P1 would show "excess position error" by way of the status byte.
Craig
Good point, the P2 doesn’t need to actually know the position especially when the leg is transitioning. I have been using that data for troubleshooting and its helped me find some issues. The P2 ignores all the position data until the leg is actually stopped and since this all runs on a separate cog, it doesn’t affect anything else. I’ve actually gotten to a point where all that extra communications is starting to get in the way so this is a good reminder to par it down some anyway.
I’m still mulling over how to get all 3 leg motors to move the leg tip so that each motor stops more or less at the same time. Since the motor operations are independent of each other, motor speeds are different, and on running on separate cogs, coming up with a way to synchronize them has been occupying my thoughts.
One idea is to compare the initial position error values for each motor (shows how far each motor has to move the leg to get to the requested
angle). If for instance the coxa has an error of 200, femur error of 25, and tibia error of 100. Then the femur moves 1 error value for every 4th coxa error value, and the tibia moves 1 error value for every 2nd coxa value (I hope I’m explaining this right!). That gives a way to know how much each motor has to move in relation to the other motors. However each motor doesn’t run at the same speed or even have the same effect on the leg tip at the same rate. For instance, the coxa motor has the fastest impact on leg tip positioning as it can move the leg tip from 0º to 180º in 2-3 seconds (I run this motor at 1/2 speed normally just for control). The tibia motor moves the leg in an out from 0º to 45º but that takes about 10 seconds. The femur moving the leg up and down has the slowest motion of all, 30º to 120º takes around 40 seconds. Normally the leg tip movement commands should be confine the leg tip to moving no more than 10mm at a time. It may be possible to just make the assumption for these very short movements, each motor is more or less synchronized as the PID isn’t going to running any motor full out over the short distance. If that is a valid assumption then I can just forget about everything I wrote in this paragraph!
If the above is true, then if the P2 creates an array of movement commands for a single step that is sent to the leg controller for storage. As the leg mostly completes a movement of all 3 motors, pop off the next movement command on the array stack, and continue the movement before the motors actually stop moving on the previous move. This could result in a smoother transition between all the individual step increments. I will have to work on coming up with some code to test this theory.
@msrobots said:
Your P2 sends commands to the legs to move. The legs move sending results.. I think this is too complicated. You should turn this upside down.
Say you have a P2 Hub array with all needed parameters and return values for each leg.
On each leg P1 you also have a HUB array with parameter and return values.
Your serial comms from P2 to P1es just sends each leg its Params and receives current return values writing back into P2 HUB. rinse repeat.
On P1es receive new params, write to own Hub read own Hub write current return values. rinse repeat.
This is as simple as it gets. Send X Hub longs to leg, receive X longs from leg, repeat. constantly. even if nothing needs to move.
On the P2 you need no command encoding, sending, reading ... just read you own HUB variables and write your own Hub variables
On the P1 you need no command encoding, sending, reading ... just read you own HUB variables and write your own Hub variables
And then let each leg P1 figure out the movement, give some start and target sysclocks for the whole movement in the HUB array or such.
On the P2 you now can calculate a movement array with proper spacing and - just write into his own Hub, the legs will go there.
think of it as common shared Hub memory between each P1 and the P2.
I’m not sure I fully understand what you are saying? Current setup has the P2 sending movement commands to each leg controller when it wants it to move, it doesn’t send anything else until the next movement is needed. The P1’s are always sending the P2 data on current position but other than showing this during debugging, the P2 ignores the return data until it’s ready to send the next movement packet.
With the earlier discussion this may change into the P2 sending batches of movement commands to the leg controllers but the leg controllers won’t respond back unless there is an error or the movement has been completed.
OMG! I've just discovered this thread. It's been around for over 10 years and I haven't noticed.
First, I have to say that all the machining and mechanical work is excellent. I love it. Really great work.
But I think you made a major mistake: It's way too big any way too complex. Don't understand me wrong. I know that reasearch and trying new things is fun even if it fails. But with such a big robot it's going to be really expensive if something turns out not to work as expected. I think the whole drive chain is totally underpowered and overweighted. To walk with reasonable speed you need to extend a leg with >1ft/s and you have to be able to push the whole thing upwards against gravity while standing on only 3 feet. On steep terrain it could happen that even a single leg has to support more than half of the weight.
Developping everything at once from scratch is a nightmare. I think it should be done step by step with a smaller model. I'd start with standard model servos. A single P2 or even a P1 can control 6 x 3 = 18 of those. You can figure out the math to coordinate all legs simultanously. If you want to climb stairs or walk on uneven terrain you have to have sensors that can "feel" the resistance of the ground. Either with force sensors in the foot tips or by measuring the load = current in the motors. I think the algorithms for motion control and terrain adaption is the biggest challange.
As soon as the smaller model works you can scale that up. I'd simply try to build bigger and stronger servos that respond as quickly as possible to a PWM or analogue command. Buffered communication doesn't work, trust me. It has to happen all in real time. As a rough estimate plan for one order of magnitude of speed for each level in the command chain. Roughly speaking, if you want to make one step per second, your position controller needs to have a reaction time of less than 0.1s. Your control loop for the motor velocities should have an update rate or ~10ms and your current amplifier around 1ms or better.
I think those linear actuators you use are designed to open/close windows or to adjust backrest or height of chairs. They are not made for continous operation and will fail quickly if they have to move all the time under load. I'd use some brushless industrial servos to drive ball screws. They cost a little more (something around $100 per 100W motor) but they are powerful, efficient and last long. Check this
Hello ManAtWork, I’m surprised you haven’t seen this thread already!
If you haven’t read all the posts over the years, this is my 2nd hexapod, the first one was less than half the size and was never close to being completed as I didn’t have the tools or knowledge necessary to finish it up. This version was started several years after that failure and I spent 1 year just testing leg mechanisms and other aspects of the project before i was satisfied enough to order the raw materials needed to create this version. It took a couple of years to build the majority of the hexapod and put it through it initial paces. It was not designed to be fast but to have a relatively slow gait. The linear actuators are each rated for moving 300 lbs but will be each sharing a portion of the almost 200 lb total weight. The linear actuators use a screw type mechanism so there is no power drawn unless actually moving. In my testing, the actuators have been strong enough to raise and lower the entire weight without any issues. On more than one occasion I’ve had a single actuator raise the robot when I made errors in coding.
Software development has been the focus for quite a while. There have been several software versions that ended up to be dead ends but each one was a learning experience. Currently there is one P1 controller for each leg running 3 motors using a modified PID controller. I did take a long break until the P2 came out as I was trying to use a P1 to control everything but it didn’t have the horsepower I needed at the time.
If I made a major mistake in the design, it’s a bit too late to making big changes at this point. I have looked into using brushless motors but haven’t found many with the power needed and at a price point I can afford (there are 18 motors that would need changing out along with new motor controllers for each one). I know there are limitations associated with using the brushed motors but I’m not planning on this winning any races so taking it slow is the name of the game. Besides, at this point I’m still working on coordinating all 3 axis motors to synchronize together. Once it actually is able to walk, I’m sure I will be introduced to other issues to solve.
This robot is a fun hobby where I wanted to challenge myself. I’ve had the opportunity to learn lots of new stuff (like CNC programming, metal milling, metal lathe, how to anodize aluminum, P1 and P2 programming, getting microcontrollers to communicate together, Inverse Kinematics mathematics, learning to use lots of new tools). I don’t regret the path I have taken, even the wrong turns have been educational. This thread has turned into more of a blog of the progress of the robot, both the good and bad parts.
Hey Bob,
Just throwing this out there but NOT suggesting that you switch direction:
The miniature hydraulics that I've been looking at on Aliexpress have piqued my interest. I have much experience with closed-loop hydraulics as well as electrics.
Ironically, I hold the patent on the 100%-electric CNC tube bending machine and I have dozens in production around north America. Today, all the big manufacturers have gone this way.....I was never a fan, however
Still can't beat hydraulics for compactness/robustness and positioning is a no-brainer. Ideally, I would love to come-up with an integrated unit where the servo-motor drives a small pump for each cylinder. This solution already exists but is needlessly expensive. I came across a Japanese machine-tool that uses this method and it blew me away; silent, compact, no excess heat, precise and clean. No gear-train and therefore no backlash to compensate for.
@Mickster said:
The miniature hydraulics that I've been looking at on Aliexpress have piqued my interest. I have much experience with closed-loop hydraulics as well as electrics.
Ironically, I hold the patent on the 100%-electric CNC tube bending machine and I have dozens in production around north America. Today, all the big manufacturers have gone this way.....I was never a fan, however
Still can't beat hydraulics for compactness/robustness and positioning is a no-brainer. Ideally, I would love to come-up with an integrated unit where the servo-motor drives a small pump for each cylinder. This solution already exists but is needlessly expensive. I came across a Japanese machine-tool that uses this method and it blew me away; silent, compact, no excess heat, precise and clean. No gear-train and therefore no backlash to compensate for.
Hydraulics was an option that I checked out when I first started the project but there were few options for small scale and it was all hideously expensive! So I kept with what I knew!
Haven’t seen much going on in the Robotics section of the forum and I haven’t been contributing for a while either. Unfortunately we had to deal with a fire in our motorhome while driving, dealing with insurance after it got totaled and flying to Vegas to pick up a replacement motor home. We then had an accident driving it home (rear ended by a semi on the interstate). Luckily no one was hurt in either accident and the new RV was not damaged too badly (hopefully repair parts will be in by February). So that and lots of other things have been taking up the majority of my time instead of getting down to the basement to spend quality time on the robot!
I accidentally messed up the robot battery pack and I don’t believe it is recoverable. The charger was left plugged in to the robot without the charger being plugged into the wall for several days. Since the charger plug is wired directly to the battery, I think it slowly drained the power out of the battery by back feeding through the charger, slow enough that the Battery Management System didn’t trigger to keep the cells from dropping below critical levels. I have a couple of tests to do but it looks like it will have to be replaced.
For replacement I am going to go with a Lithium Iron Phosphate battery vs the original Lithium Ion. LiFePh batteries are slightly less energy dense as LiIon but more importantly, they aren’t as big a fire risk as Lithium Ion. Large Lithium Ion battery packs tend to make me nervous during charging since that is when they are most susceptible to fire. Since I put together the original battery there are now many other battery size options out there at reasonable cost.
Although I haven’t spent much time with the robot, I have been able to get some time on it, mostly reviewing code, trying to improve some of the more dodgy coding, and trying out suggestions that people have made. I looked more into the communications schema I’m using between the Propellors for ways to improve it. The way comms are set up now, comms run on their own cogs so the speed of the comms doesn’t impact the main software loops. The P1 transmitting comms read the current value of various global variables, assemble the data and push it out. The P2 transmission is in its main software loop but the P2 runs so much faster than the P1’s, that the P2 can send dozens of commands before the individual legs can actually react. I did make a few changes to simplify a few areas in the comms that seem to be working OK so far.
Another area I have been looking into is the suggestions about how to better synchronize individual leg movements. Although the P2 sends out the movement command for all 3 motors at the same time, the actual motor reaction times can be quite different. Each motor operates through a series of linkages and may have a short or long movement required based on the input command. Each motor operates in its own cog with the only feedback being the magnetic angular encoders. A motor/linkage may also operate faster or slower depending on the direction of movement as more torque is required if moving the weight of the body vs just moving freely. Now the goal here is for all 3 motors to start moving at the same time (this mostly happens as each motor cog gets their movement command at the same time) and for each movement to be completed at the same time (that’s the tricky part). I think I need another cog that looks at the distance each motor has to travel, adjusts the speed of 2 faster motions to the speed of the slowest motor. So far I’ve come up with a blank on how to do this.
Once I save enough I can order a new battery and I can get back to work, in the meantime the robot isn’t moving without power!
@DiverBob said:
Another area I have been looking into is the suggestions about how to better synchronize individual leg movements. Although the P2 sends out the movement command for all 3 motors at the same time, the actual motor reaction times can be quite different. Each motor operates through a series of linkages and may have a short or long movement required based on the input command. Each motor operates in its own cog with the only feedback being the magnetic angular encoders. A motor/linkage may also operate faster or slower depending on the direction of movement as more torque is required if moving the weight of the body vs just moving freely. Now the goal here is for all 3 motors to start moving at the same time (this mostly happens as each motor cog gets their movement command at the same time) and for each movement to be completed at the same time (that’s the tricky part). I think I need another cog that looks at the distance each motor has to travel, adjusts the speed of 2 faster motions to the speed of the slowest motor. So far I’ve come up with a blank on how to do this.
When you need to move several motors simultanously in a way that the tip of the leg has to follow a well defined trajectory it is not sufficient to only command the target position and then let them move independently. As you already figured out, even if you also command a target velocity to each motor so that they theoretically should reach the target position at the same time thigs can go wrong if there is resistance or the kinematics are non-linear and one of the motor falls behind.
You need to plan the trajectory ahead of the time and put a series of position vectors in a look-ahead buffer. Say you calculate a position vector for every 5ms of movement. Then you execute the coordinated movement by sending a position commands to each of the involved axes every 5ms. Then you monitor the actual positions. If the following error of one of the motors goes above a certain threshold you need to slow down the other motors as well by stretching the time-scale so that every loop iteration takes longer than 5ms.
@ManAtWork Yup, it's all in this thread. It's actually one of the easiest things to handle. It just gets ignored.
Just like a movie is a series of still images, that's exactly how motion is controlled.
Axis #1 Target Position: 90 deg
Axis #2 Target Position: 45 deg
Both axes required to arrive at exactly 1000ms (vector velocity)
Command update rate: 5ms (can be much greater or smaller)
Number of updates = 1000ms/5ms = 200
For each 5ms update, Axis #1 command position is incremented by 90deg/200
For each 5ms update, Axis #2 command position is incremented by 45deg/200
Craig
Incidentally, according to a Kuka-Robotics engineer, their update period is 14ms (bit of a weird number)
Yes, I think something in the range of 5..10ms is reasonable. I use 1.25ms for my CNC controller where the tool needs to precisely follow a path with tight tolerances. But that's overkill for a robot. 20ms is too much, it causes vibrations.
If you do a simple linear interpolation of position vs. time (see craig's example above) you don't need a lookahead buffer and could calculate the positions vectors on the fly. But it's better to slow down before you reach the target position, otherwise the position will overshot due to inertia. This is why you need to look ahead.
It’s been a while since I have been able to work on the robot, ended up spending 4 weeks in Florida to get away from the cold Michigan winter.
It took a while to locate a battery that could fit into the existing space and even then I had to relocate the 12 to 5 vdc power supply to make room. I got a 12 volt, 20 amp-hr battery, a bit smaller than I would like but I’ll see how long it lasts during testing.
So ready to get back on the robot programming and put some of these ideas to work!
Thanks Erco! Spent yesterday getting the programming computer up to date with Windows updates, Spin IDE updates, backups and virus protection all running again. While that was running I spent some time re-reading some of the later comments posted on the thread to figure out where I left off and suggestions from others.
Mickster and ManAtWork had a suggestion about using timed movements instead of discrete 10mm movements. Now I send out a move command and wait for the move to complete before sending out the next movement command. As I interpret what they are saying, use discrete time periods to make that 10mm move. So I could calculate the movements for all 6 legs and send the first 10mm move command out simultaneously to each leg. When a fixed unit of time goes by (to be determined experimentally but say 15ms for example), send out the next set of 10mm movements to the legs regardless of actual leg axis position. Repeat this until the legs reach the final destination. Thinking it over this could solve several issues, even if a movement is too small to exceed the axis deadband or the leg hasn’t reached the desired location, over time the error will accumulate enough to force movement. By not waiting for each motor to complete a movement, if the timer is set correctly the leg axis should move on to the next movement without having to stop. This will be fun to figure out how to implement, I wonder if setting up a spare cog on the master as a dedicated timer might work, have it set a flag or counter whenever the time interval is hit so the master knows its time to release the next set of movement commands.
Another suggestion was to calculate all the individual movement commands up front, store them in an array and pop them off the stack when the timer activates. If the master takes longer to complete the IK calculations than the time interval then this would be the way to go. Otherwise, if the P2 can perform all the calculations in less time than the timer, calculations can be done as needed. I need to run a time trial on the current calculation process, I believe I’ve seen some code somewhere that makes it easy to time your code, just have to find it again.
Looks like I have some more testing to do!
A pretty cool thing that I have going is a BASIC interpreter/editor right on the device (STM32H743 @480MHz). Code execution is similar to SPIN on P1 but I can simply edit/run without the download.
I am running a servo, no sweat. Both the PID and the position update happen in the same 1Khz loop. I stripped-out the irrelevant BASIC code so this is like pseudo-code:
do
[waitcnt 1ms]
actpos=[read axis actual position]
poserr=cmdpos-actpos
motcmd=(kp*poserr)+(kd*(poserr-preverr))+(ki*integrator)
preverr=poserr
a=inkey$
if len(a) then
if a="m" then inc spdcmd,1 [keyboard command "m" = more velocity]
if a="l" then inc spdcmd,-1 [keyboard command "l" = less velocity]
end if
inc cmdpos,spdcmd
loop
"spdcmd" is encoder counts. Obviously, in this case, each increment of spdcmd = 1000 cts/sec which is just for clarity
spdcmd is where you put your target position divided by the loop-rate (1000 in this case)
Point being, put everything in the PID loop. Externally, you'll need to scale the commands and take care of velocity profiling etc.
But it doesn't get much better than this. 1KHz is more than adequate for pretty much any mechanical device and here it is running on an interpreted language.
I have much more going on than shown above but of my 1ms window, I'm a little under 250usec.
Err = Following Error
Vel = Actual motor velocity (500 line encoder, 2000 quadrature counts/rev)
The other number is execution time. The P1 @80MHz should be similar with SPIN.
I've started experimenting with using a timer to coordinate the 3 motors vs waiting for each motor to complete its movement.
I decided to first try implementing the clock on the master P2 instead of the individual P1 leg controllers. The reason behind this is the P2 is the source for all movement commands and I can more easily keep each leg in sync with the other legs by regulating the timing of when the P2 outputs a movement command. The P1 control software already processes an incoming P2 command even if the previous command has not completed. I'll try this method out first and see what kind of results I get.
All the coding is being done on the P2 master at this point. I set up a clock that gets incremented every 1 millisecond. Since I didn't know how fast the test routine ran, I timed the main loop in the program that is outputting data to the individual leg controllers. I found the loop was running about 142 usec, so I have lots of time available to play with.
Initial testing is with a single leg, telling it to move at a 45 degree angle for 200 millimeters. I set up a delay interval of 25 milliseconds just as a starting point. The leg visually seemed to be moving fine and then suddenly moved much faster to a final position. Review of the Debug logs shows the leg didn't actually start moving until the 6th movement command. Then the leg movement in all axis were running well behind the incoming command positions and the difference between commanded position and actual position kept getting larger. The final burst of movement I was seeing was the motors finally catching up the final position.
I increased the time interval to 1 second just to see how the motors responded, even there the motors were not keeping up with the commanded position. Plus, the final position the leg ended up in isn't where I expected.
So it looks like I made an conceptual error in the way I thought the leg movement should be calculated since the leg isn't getting to the expected position (drawing this out on paper I think I didn't account for leg position offsets). The fact the leg is just starting to move after the 5th or 6th command set means there is some backlash between the motor and the axis it is trying to move.
Got some more thinking to do!
Yup, PID doesn't like backlash. This is actually a pet-peeve of mine because the industry has kinda gone backwards. Most machinery, today, relies on the feedback device that is fixed to the motor shaft. I had a client call me out on what he believed to be a process problem (metal forming). He didn't question the control system because the display showed zero error. Well the motor shaft was where it had been commanded to be but it was totally oblivious to the 2mm of backlash in the drivetrain.
Many years ago, I came across some opposition from a particular mafia group of engineers within a large automotive company. They were totally opposed to the idea of me rebuilding and re-controlling their older machines. At the time, it seemed like every engineer was obsessed with six-sigma, continuous improvement, yada, yada and an old machine could never be as accurate/repeatable as a new machine. I knew what they were really concerned about which was the fact that they got a week of wining/dining/adult-venues on the West coast when they went to the manufacturer to sign-off on a new machine, prior to delivery.
So I threw down the gauntlet and guaranteed them that my rebuilds would out-perform any of their new equipment. Of course they laughed and gleefully accepted the challenge. What they didn't know was that I use dual-loop feedback. Motor feedback for axis stability and an auxiliary feedback device mounted on the actual load.
They went to great lengths to produce an apparatus involving their own bolt-on feedback devices connected to a data-acquisition PC (they didn't do this for new machines). The nominal spec was a repeatability of +/- 0.1mm (kinda sloppy by our standards) and so their external devices were good for +/- 0.05mm (the intention was to humiliate me). Well my "Sony Digi-Rulers" were good for +/- 0.01mm which was what we could hit, every single time
Their software was supposed to collect variable data and plot bell-curves etc. Imagine the look on their faces when they weren't able to record any error whatsoever. "How are we supposed to plot a bell-curve if we don't have any variable data?" Not my problem
I respectfully requested that we perform the same experiment on their newest machine but somehow they never got around to it...I got the contract
Having position on both the motor and load is a great way to really know what is going on. My system has the sensors on the load (the moving part of the leg) but nothing on the motor as the linear actuators are sealed units. I was looking into putting some type of optical reader inside the housing but there isn’t any real room (there is a 10 turn potentiometer inside the housing but it reads off the output shaft so not very helpful in this case.
I did a layout of the leg movement on paper and found a couple of areas I could improve the code and hopefully get better accuracy in the movement. I’m still considering options for dealing with backlash. Since I have an indication of when the load starts to move, I could send an initial movement command but not start the timer until feedback indicates the load is actually moving. This approach could get complicated since in some cases one or more leg axis movement command might be smaller than the deadband value.
Anyway, that’s half the fun of this project is finding a problem and then figuring out a solution that fits both my abilities and budget! First step is to code a example using the new ideas and see how that works out, hopefully not introducing too many new issues…
Methinks that the backlash between the actuator and the feedback has necessitated a low-ish proportional gain (Kp) to achieve stability, which could also explain the lag.
A useful trick is to apply an offset to the command, external to the PID loop. This offset will take-up the backlash, prior to the PID doing its thing (PID only responds to error so give it a push to minimize the error).
@Mickster said:
Methinks that the backlash between the actuator and the feedback has necessitated a low-ish proportional gain (Kp) to achieve stability, which could also explain the lag.
A useful trick is to apply an offset to the command, external to the PID loop. This offset will take-up the backlash, prior to the PID doing its thing (PID only responds to error so give it a push to minimize the error).
Am I interpreting this correctly: if a movement command is received that reverses motor direction (I’m assuming that continuous moves in the same direction have already taken up any backlash), outside of the PID initialize a a higher error value to get the motor starting up faster than the normal PID loop would allow? Or actually start the motor up and moving before the PID loop starts up?
And this is already calibrated for the optimum response/stability (limited, due to backlash)
Add an offset to provide a bias in the direction of intended motion
Motor_Command = Kp+Kd+Ki+Offset
If the offset is set a little too high, the Kp+Kd+Ki will oppose it like it was a load disturbance so it won't run away
Using a new method for moving a leg in a straight line, I coded and tested the new setup. I also put in the ability to send the movement commands from the P2 to the P1 controllers at discrete intervals.
The method assumes that the user will input the direction of travel and the length of the stride of the leg. To keep things simpler, the height of the robot is fixed. Individual legs are assigned positions on an XY grid with the 0,0 point being the center of the robot and the legs located at 60º intervals in a circle around the grid. I assigned leg 1 to be located at 0º, so if the user puts in a travel direction of 45º, leg 1 will also move 45º, leg 2 is offset by 60º so its movement angle is 45º - 60º or 15º (based on the local leg 2 coordinates). Working with leg 1 to start (easier calculations and easier to visually see if something isn’t going right), I break up the distance into discrete intervals (using 5mm as a starting point). Dividing the total stride length by the interval gives the number of individual positions the leg will move. Next is determining the amount of X and Y movement needed for these discrete moves. If starting with a 45º angle with a length equal to the interval (5 mm = hypotenuse), the cordic comes back with the X and Y value change needed for each individual movement. The change in X and Y is added to the current X and Y position and the leg moves to the new position.
Previously the coding waited for each motor to finish its move before the next move command was sent. This results in a very jerky movement. As suggested by Mickster, I added a timer that controls when the movement command is released. This value is being determined experimentally (35 msec is looking good). Too long an interval and the leg move gets jerky, too short and the motors can’t keep up.
Current problem I’m also working through is backlash. During testing I reset the leg to a known position and then have it move to the desired location. What happens here is that at the start of the movement the motor starts turning but it’s about 250 msec before the leg sees any physical movement from the sensors. I tried a quick idea of adding an extra 250 msec to the first movement timer but that didn’t have the desired effect. Mickster recommended adding an offset to the PID loop in the PID code when a motor direction change is noted. I got back into the leg controller code and quickly realized earlier I had changed the PID quite a bit so that the femur and tibia were modified that it wasn’t really a PID anymore as the motors were just told to run one direction or another until they reached a desired position. The femur and tibia both use linear actuators which are fairly slow moving devices and the distances they would be asked to move would be fairly small so locking in the motor speed was the best solution. The coxa still uses the PID as it responds much more quickly, there I can experiment with an offset value. I was able to speed the motors up some. Next stage is uploading the change to the leg controllers and see how much of a difference that makes.
I would like to post some video of the leg movements but don’t know how to upload short video clips to this forum without going through YouTube. Is it possible to upload 10-15 second long video here?
Hi,
Just an idea. To get some more information it would be nice to know the actual speed of a motor. So did you consider to measure back electromotive force? This might be helpful because you actually have this speed sensor for each axis for free on the motor shaft.
Christof
Yeah, as noted by @ManAtWork, that 35msec rate would normally be jerky but it's clear that your overall loop gain is low (sluggish) in order to tolerate the lost motion (backlash).
Sounds like you're on the right track though except for going open-loop on the femur and tibia....
the motors were just told to run one direction or another until they reached a desired position
Once you get to grips with the discretized commands and offsets, I reckon that you could reinstate the PID.
Theoreticians tend to assume a perfect world and so pretty-much everything published, regarding PID is totally oblivious to reality. I have many hundreds of axes using a "totally unsuitable" (a PhD from Vickers-Rexroth) proportional valve. He was right.... horrendous dead-band but I got creative with the PID and arriving at +/- 1 encoder count became a breeze. I'd previously used a high-end servo-valve but even with 10 micron oil filters, they still failed (big bucks). These proportional valves, I could put sludge through them and they'd keep on trucking.
@"Christof Eb." said:
Hi,
Just an idea. To get some more information it would be nice to know the actual speed of a motor. So did you consider to measure back electromotive force? This might be helpful because you actually have this speed sensor for each axis for free on the motor shaft.
Christof
Now that he has a sample-time, he pretty-much has it; dp/dt=velocity.
Made the updates to the leg controller program to increase motor speeds under specific conditions. Initial testing shows the response time for the femur to reverse motor direction and take up backlash has been reduced to about 100 milliseconds vs 250 milliseconds. Testing shows that 35 millisecond loop time is where the motors start to jerk due to coming to a stop before the next command is received. 30 milliseconds seems to be the sweet spot with the current setup. I added a backlash timer that runs if any motor reverses direction (meaning there is backlash to take up) so the initial movement timer is increased to 50 milliseconds. This idea seems to be a work in progress.
I attached a copy of one of the debug logs and replicated some of first data it is giving me.
Cog0 Test 5x - move single leg in straight line using IK and timed movement
Cog0 Transmited command: $,1,828,75,900,1
Cog0 Leg 1, legXActual[n] = 0, legYActual[n] = 297, legZActual[n] = 610
Cog0 -------- coordinateConversion ----------
Cog0 Leg 1: moveAngle: 450
Cog0 Leg 1, bodystride = 200, legangle[n] = 450, resolution = 5, stepincrement = 40
Cog0 Start position for Leg: legXActual[n] = 0, legYActual[n] = 297, legZActual[n] = 610
Cog0 Step distance: x = 4, y = 4, z = 0
Cog0 Loop interval (msec): msecDelay = 35
Cog0 ---------- Start timed movement ----------
Cog0 Step: 1 Next Position Leg 1, legGaitX[n] = 4, legGaitY[n] = 301, legGaitZ[n] = 610
Cog0 Transmited command: $,1,827,76,892,1
Cog0 Leg 1, legXActual[n] = 4, legYActual[n] = 297, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 814 Tibia: 53 Coxa: 962
Cog0 Position error: femurAngle[n]-fangle = -13, tibiaAngle[n]-tangle = -23, cangle-coxaAngle[n] = -70
Cog0 ------- End Step -------
Cog0
Cog0 Step: 2 Next Position Leg 1, legGaitX[n] = 8, legGaitY[n] = 305, legGaitZ[n] = 610
Cog0 Transmited command: $,1,831,79,884,1
Cog0 Leg 1, legXActual[n] = 8, legYActual[n] = 300, legZActual[n] = 611
Cog0 Leg 1 angle values: Femur: 814 Tibia: 54 Coxa: 962
Cog0 Position error: femurAngle[n]-fangle = -17, tibiaAngle[n]-tangle = -25, cangle-coxaAngle[n] = -78
Cog0 ------- End Step -------
Cog0
Cog0 Step: 3 Next Position Leg 1, legGaitX[n] = 12, legGaitY[n] = 309, legGaitZ[n] = 610
Cog0 Transmited command: $,1,831,82,877,1
Cog0 Leg 1, legXActual[n] = 12, legYActual[n] = 303, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 816 Tibia: 54 Coxa: 962
Cog0 Position error: femurAngle[n]-fangle = -15, tibiaAngle[n]-tangle = -28, cangle-coxaAngle[n] = -85
Cog0 ------- End Step -------
Cog0
Cog0 Step: 4 Next Position Leg 1, legGaitX[n] = 16, legGaitY[n] = 313, legGaitZ[n] = 610
Cog0 Transmited command: $,1,835,88,870,1
Cog0 Leg 1, legXActual[n] = 16, legYActual[n] = 311, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 816 Tibia: 59 Coxa: 958
Cog0 Position error: femurAngle[n]-fangle = -19, tibiaAngle[n]-tangle = -29, cangle-coxaAngle[n] = -88
Cog0 ------- End Step -------
Cog0
Cog0 Step: 5 Next Position Leg 1, legGaitX[n] = 20, legGaitY[n] = 317, legGaitZ[n] = 610
Cog0 Transmited command: $,1,838,92,863,1
Cog0 Leg 1, legXActual[n] = 20, legYActual[n] = 315, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 824 Tibia: 65 Coxa: 958
Cog0 Position error: femurAngle[n]-fangle = -14, tibiaAngle[n]-tangle = -27, cangle-coxaAngle[n] = -95
Cog0 ------- End Step -------
I attached the debug log from one of the test runs. The leg is moving at 45 degrees for 200mm. The individual step interval is 5mm so it takes 40 steps to travel that distance. The leg X,Y,Z position values are in mm. The Position Error shows how much difference there is between the requested angle and the actual angle of the leg during the step. I copied the first 5 steps where you can see the femur angle starts at 81.4 degrees (814 due to using integer math), the P2 wants the femur to move to 82.7 so there is a difference of 1.3 degrees. The femur stays at 814 until step 3 where it increases to 816 but stays there for the following step also (this is taking up backlash). By step 5 the femur is now moving but doesn't catch up. By step 32 the femur starts to fall behind progressively more each step. The tibia tends to follow the requested movement fairly well, only about a single degree difference. Coxa motor wanders faster and slower but maintains itself around 4 to 5 degrees behind the needed position. the very last 2 lines of the log show the final stopping point. all motors are only 2 degrees off from the final position.
I'm going to play with the backlash and timer interval values more along with the step interval distance, I've got some other ideas to try out based on some of the input from Mickster and some research I've uncovered.
@"Christof Eb." said:
Hi,
Just an idea. To get some more information it would be nice to know the actual speed of a motor. So did you consider to measure back electromotive force? This might be helpful because you actually have this speed sensor for each axis for free on the motor shaft.
Christof
Now that he has a sample-time, he pretty-much has it; dp/dt=velocity.
Well, as there is so much backlash, he does not have it for a lot of time. Using the motor as a tachogenerator works instantly and surprisingly well with DC motors.
I ran into a problem during testing that had been very infrequent but lately happens almost every time I download a program update to the P2 controller. There are random leg movements from legs that aren't being sent a movement command. This seems to be specifically happening during the initial P2 setup routine and right after the ports for the leg have been initialized. It’s acting like the buffer has a valid command when nothing has been sent yet. Movement that isn't requested is a serious problem, luckily the robot is on a test stand with the legs not touching the ground so no damage so far. But I just had leg 6 move the coxa motor over far enough for it to interfere with leg 5 when it tried to move.
EDIT: I’ve been fixating in the idea that the P2 was randomly sending a valid movement command to the P1’s and I couldn’t find a source of the signal. I dug into the original HB-25 motor controller docs and found it only takes a single pulse to the controller in the range of 1-2 milliseconds to initiate a continuous PWM output. This corresponds with the movements have been for the full travel of the affected axis and only stop when they reach full travel. It seems likely that the HB-25 control driver is occasionally outputting a valid pulse while the P2 is initializing the communications ports.
Something else to look into today!
EDIT 2: Ever had a day when everything goes bad? Today was that day for me. I made what seemed to be fairly simple code changes, uploaded the changes and actually made the situation much worse! Trying to figure out why the changes were causing this much additional issues has been difficult and time consuming. I’ll roll back the updates tomorrow and try again.
Comments
Following your progress since years and love it.
I was thinking and sometimes that is dangerous. It is about your communication method.
Your P2 sends commands to the legs to move. The legs move sending results.. I think this is too complicated. You should turn this upside down.
Think alike the mailbox system used in spin to cog communication.
Decoupling for parallel execution not serial. take the comms out of the way of thinking.
Say you have a P2 Hub array with all needed parameters and return values for each leg.
On each leg P1 you also have a HUB array with parameter and return values.
Your serial comms from P2 to P1es just sends each leg its Params and receives current return values writing back into P2 HUB. rinse repeat.
On P1es receive new params, write to own Hub read own Hub write current return values. rinse repeat.
This is as simple as it gets. Send X Hub longs to leg, receive X longs from leg, repeat. constantly. even if nothing needs to move.
On the P2 you need no command encoding, sending, reading ... just read you own HUB variables and write your own Hub variables
On the P1 you need no command encoding, sending, reading ... just read you own HUB variables and write your own Hub variables
And then let each leg P1 figure out the movement, give some start and target sysclocks for the whole movement in the HUB array or such.
On the P2 you now can calculate a movement array with proper spacing and - just write into his own Hub, the legs will go there.
think of it as common shared Hub memory between each P1 and the P2.
Sure you have some Latency, but that latency is constant since your serial comms to and from leg runs all the time the same code without stopping or changing.
Speed (velocity) = dp/dt
Increase dp and we increase velocity.
We command each joint (motor) to be at a certain position at a certain time and voila, we have synchronisation, courtesy of the P1s' clocks.
P2 doesn't need to constantly monitor the positions of each joint. Whatever it last commanded, should be the actual position. If not, the P1 would show "excess position error" by way of the status byte.
Craig
Good point, the P2 doesn’t need to actually know the position especially when the leg is transitioning. I have been using that data for troubleshooting and its helped me find some issues. The P2 ignores all the position data until the leg is actually stopped and since this all runs on a separate cog, it doesn’t affect anything else. I’ve actually gotten to a point where all that extra communications is starting to get in the way so this is a good reminder to par it down some anyway.
I’m still mulling over how to get all 3 leg motors to move the leg tip so that each motor stops more or less at the same time. Since the motor operations are independent of each other, motor speeds are different, and on running on separate cogs, coming up with a way to synchronize them has been occupying my thoughts.
One idea is to compare the initial position error values for each motor (shows how far each motor has to move the leg to get to the requested
angle). If for instance the coxa has an error of 200, femur error of 25, and tibia error of 100. Then the femur moves 1 error value for every 4th coxa error value, and the tibia moves 1 error value for every 2nd coxa value (I hope I’m explaining this right!). That gives a way to know how much each motor has to move in relation to the other motors. However each motor doesn’t run at the same speed or even have the same effect on the leg tip at the same rate. For instance, the coxa motor has the fastest impact on leg tip positioning as it can move the leg tip from 0º to 180º in 2-3 seconds (I run this motor at 1/2 speed normally just for control). The tibia motor moves the leg in an out from 0º to 45º but that takes about 10 seconds. The femur moving the leg up and down has the slowest motion of all, 30º to 120º takes around 40 seconds. Normally the leg tip movement commands should be confine the leg tip to moving no more than 10mm at a time. It may be possible to just make the assumption for these very short movements, each motor is more or less synchronized as the PID isn’t going to running any motor full out over the short distance. If that is a valid assumption then I can just forget about everything I wrote in this paragraph!
If the above is true, then if the P2 creates an array of movement commands for a single step that is sent to the leg controller for storage. As the leg mostly completes a movement of all 3 motors, pop off the next movement command on the array stack, and continue the movement before the motors actually stop moving on the previous move. This could result in a smoother transition between all the individual step increments. I will have to work on coming up with some code to test this theory.
I’m not sure I fully understand what you are saying? Current setup has the P2 sending movement commands to each leg controller when it wants it to move, it doesn’t send anything else until the next movement is needed. The P1’s are always sending the P2 data on current position but other than showing this during debugging, the P2 ignores the return data until it’s ready to send the next movement packet.
With the earlier discussion this may change into the P2 sending batches of movement commands to the leg controllers but the leg controllers won’t respond back unless there is an error or the movement has been completed.
No "more or less" about it. We're talking sample accurate. Late (early) here. Will reply in a few hours.
Craig
OMG! I've just discovered this thread. It's been around for over 10 years and I haven't noticed.
First, I have to say that all the machining and mechanical work is excellent. I love it. Really great work.
But I think you made a major mistake: It's way too big any way too complex. Don't understand me wrong. I know that reasearch and trying new things is fun even if it fails. But with such a big robot it's going to be really expensive if something turns out not to work as expected. I think the whole drive chain is totally underpowered and overweighted. To walk with reasonable speed you need to extend a leg with >1ft/s and you have to be able to push the whole thing upwards against gravity while standing on only 3 feet. On steep terrain it could happen that even a single leg has to support more than half of the weight.
Developping everything at once from scratch is a nightmare. I think it should be done step by step with a smaller model. I'd start with standard model servos. A single P2 or even a P1 can control 6 x 3 = 18 of those. You can figure out the math to coordinate all legs simultanously. If you want to climb stairs or walk on uneven terrain you have to have sensors that can "feel" the resistance of the ground. Either with force sensors in the foot tips or by measuring the load = current in the motors. I think the algorithms for motion control and terrain adaption is the biggest challange.
As soon as the smaller model works you can scale that up. I'd simply try to build bigger and stronger servos that respond as quickly as possible to a PWM or analogue command. Buffered communication doesn't work, trust me. It has to happen all in real time. As a rough estimate plan for one order of magnitude of speed for each level in the command chain. Roughly speaking, if you want to make one step per second, your position controller needs to have a reaction time of less than 0.1s. Your control loop for the motor velocities should have an update rate or ~10ms and your current amplifier around 1ms or better.
I think those linear actuators you use are designed to open/close windows or to adjust backrest or height of chairs. They are not made for continous operation and will fail quickly if they have to move all the time under load. I'd use some brushless industrial servos to drive ball screws. They cost a little more (something around $100 per 100W motor) but they are powerful, efficient and last long. Check this
Hello ManAtWork, I’m surprised you haven’t seen this thread already!
If you haven’t read all the posts over the years, this is my 2nd hexapod, the first one was less than half the size and was never close to being completed as I didn’t have the tools or knowledge necessary to finish it up. This version was started several years after that failure and I spent 1 year just testing leg mechanisms and other aspects of the project before i was satisfied enough to order the raw materials needed to create this version. It took a couple of years to build the majority of the hexapod and put it through it initial paces. It was not designed to be fast but to have a relatively slow gait. The linear actuators are each rated for moving 300 lbs but will be each sharing a portion of the almost 200 lb total weight. The linear actuators use a screw type mechanism so there is no power drawn unless actually moving. In my testing, the actuators have been strong enough to raise and lower the entire weight without any issues. On more than one occasion I’ve had a single actuator raise the robot when I made errors in coding.
Software development has been the focus for quite a while. There have been several software versions that ended up to be dead ends but each one was a learning experience. Currently there is one P1 controller for each leg running 3 motors using a modified PID controller. I did take a long break until the P2 came out as I was trying to use a P1 to control everything but it didn’t have the horsepower I needed at the time.
If I made a major mistake in the design, it’s a bit too late to making big changes at this point. I have looked into using brushless motors but haven’t found many with the power needed and at a price point I can afford (there are 18 motors that would need changing out along with new motor controllers for each one). I know there are limitations associated with using the brushed motors but I’m not planning on this winning any races so taking it slow is the name of the game. Besides, at this point I’m still working on coordinating all 3 axis motors to synchronize together. Once it actually is able to walk, I’m sure I will be introduced to other issues to solve.
This robot is a fun hobby where I wanted to challenge myself. I’ve had the opportunity to learn lots of new stuff (like CNC programming, metal milling, metal lathe, how to anodize aluminum, P1 and P2 programming, getting microcontrollers to communicate together, Inverse Kinematics mathematics, learning to use lots of new tools). I don’t regret the path I have taken, even the wrong turns have been educational. This thread has turned into more of a blog of the progress of the robot, both the good and bad parts.
Bob Sweeney
@DiverBob
Hey Bob,
Just throwing this out there but NOT suggesting that you switch direction:
The miniature hydraulics that I've been looking at on Aliexpress have piqued my interest. I have much experience with closed-loop hydraulics as well as electrics.
Ironically, I hold the patent on the 100%-electric CNC tube bending machine and I have dozens in production around north America. Today, all the big manufacturers have gone this way.....I was never a fan, however
Still can't beat hydraulics for compactness/robustness and positioning is a no-brainer. Ideally, I would love to come-up with an integrated unit where the servo-motor drives a small pump for each cylinder. This solution already exists but is needlessly expensive. I came across a Japanese machine-tool that uses this method and it blew me away; silent, compact, no excess heat, precise and clean. No gear-train and therefore no backlash to compensate for.
Craig
Hydraulics was an option that I checked out when I first started the project but there were few options for small scale and it was all hideously expensive! So I kept with what I knew!
Haven’t seen much going on in the Robotics section of the forum and I haven’t been contributing for a while either. Unfortunately we had to deal with a fire in our motorhome while driving, dealing with insurance after it got totaled and flying to Vegas to pick up a replacement motor home. We then had an accident driving it home (rear ended by a semi on the interstate). Luckily no one was hurt in either accident and the new RV was not damaged too badly (hopefully repair parts will be in by February). So that and lots of other things have been taking up the majority of my time instead of getting down to the basement to spend quality time on the robot!
I accidentally messed up the robot battery pack and I don’t believe it is recoverable. The charger was left plugged in to the robot without the charger being plugged into the wall for several days. Since the charger plug is wired directly to the battery, I think it slowly drained the power out of the battery by back feeding through the charger, slow enough that the Battery Management System didn’t trigger to keep the cells from dropping below critical levels. I have a couple of tests to do but it looks like it will have to be replaced.
For replacement I am going to go with a Lithium Iron Phosphate battery vs the original Lithium Ion. LiFePh batteries are slightly less energy dense as LiIon but more importantly, they aren’t as big a fire risk as Lithium Ion. Large Lithium Ion battery packs tend to make me nervous during charging since that is when they are most susceptible to fire. Since I put together the original battery there are now many other battery size options out there at reasonable cost.
Although I haven’t spent much time with the robot, I have been able to get some time on it, mostly reviewing code, trying to improve some of the more dodgy coding, and trying out suggestions that people have made. I looked more into the communications schema I’m using between the Propellors for ways to improve it. The way comms are set up now, comms run on their own cogs so the speed of the comms doesn’t impact the main software loops. The P1 transmitting comms read the current value of various global variables, assemble the data and push it out. The P2 transmission is in its main software loop but the P2 runs so much faster than the P1’s, that the P2 can send dozens of commands before the individual legs can actually react. I did make a few changes to simplify a few areas in the comms that seem to be working OK so far.
Another area I have been looking into is the suggestions about how to better synchronize individual leg movements. Although the P2 sends out the movement command for all 3 motors at the same time, the actual motor reaction times can be quite different. Each motor operates through a series of linkages and may have a short or long movement required based on the input command. Each motor operates in its own cog with the only feedback being the magnetic angular encoders. A motor/linkage may also operate faster or slower depending on the direction of movement as more torque is required if moving the weight of the body vs just moving freely. Now the goal here is for all 3 motors to start moving at the same time (this mostly happens as each motor cog gets their movement command at the same time) and for each movement to be completed at the same time (that’s the tricky part). I think I need another cog that looks at the distance each motor has to travel, adjusts the speed of 2 faster motions to the speed of the slowest motor. So far I’ve come up with a blank on how to do this.
Once I save enough I can order a new battery and I can get back to work, in the meantime the robot isn’t moving without power!
Bob Sweeney
When you need to move several motors simultanously in a way that the tip of the leg has to follow a well defined trajectory it is not sufficient to only command the target position and then let them move independently. As you already figured out, even if you also command a target velocity to each motor so that they theoretically should reach the target position at the same time thigs can go wrong if there is resistance or the kinematics are non-linear and one of the motor falls behind.
You need to plan the trajectory ahead of the time and put a series of position vectors in a look-ahead buffer. Say you calculate a position vector for every 5ms of movement. Then you execute the coordinated movement by sending a position commands to each of the involved axes every 5ms. Then you monitor the actual positions. If the following error of one of the motors goes above a certain threshold you need to slow down the other motors as well by stretching the time-scale so that every loop iteration takes longer than 5ms.
@ManAtWork Yup, it's all in this thread. It's actually one of the easiest things to handle. It just gets ignored.
Just like a movie is a series of still images, that's exactly how motion is controlled.
Axis #1 Target Position: 90 deg
Axis #2 Target Position: 45 deg
Both axes required to arrive at exactly 1000ms (vector velocity)
Command update rate: 5ms (can be much greater or smaller)
Number of updates = 1000ms/5ms = 200
For each 5ms update, Axis #1 command position is incremented by 90deg/200
For each 5ms update, Axis #2 command position is incremented by 45deg/200
Craig
Incidentally, according to a Kuka-Robotics engineer, their update period is 14ms (bit of a weird number)
Yes, I think something in the range of 5..10ms is reasonable. I use 1.25ms for my CNC controller where the tool needs to precisely follow a path with tight tolerances. But that's overkill for a robot. 20ms is too much, it causes vibrations.
If you do a simple linear interpolation of position vs. time (see craig's example above) you don't need a lookahead buffer and could calculate the positions vectors on the fly. But it's better to slow down before you reach the target position, otherwise the position will overshot due to inertia. This is why you need to look ahead.
It’s been a while since I have been able to work on the robot, ended up spending 4 weeks in Florida to get away from the cold Michigan winter.
It took a while to locate a battery that could fit into the existing space and even then I had to relocate the 12 to 5 vdc power supply to make room. I got a 12 volt, 20 amp-hr battery, a bit smaller than I would like but I’ll see how long it lasts during testing.
So ready to get back on the robot programming and put some of these ideas to work!
DiverBob: You are the most persistent, goal-oriented person ever. God bless you for your loyalty and commitment to this amazing long-term project!
Thanks Erco! Spent yesterday getting the programming computer up to date with Windows updates, Spin IDE updates, backups and virus protection all running again. While that was running I spent some time re-reading some of the later comments posted on the thread to figure out where I left off and suggestions from others.
Mickster and ManAtWork had a suggestion about using timed movements instead of discrete 10mm movements. Now I send out a move command and wait for the move to complete before sending out the next movement command. As I interpret what they are saying, use discrete time periods to make that 10mm move. So I could calculate the movements for all 6 legs and send the first 10mm move command out simultaneously to each leg. When a fixed unit of time goes by (to be determined experimentally but say 15ms for example), send out the next set of 10mm movements to the legs regardless of actual leg axis position. Repeat this until the legs reach the final destination. Thinking it over this could solve several issues, even if a movement is too small to exceed the axis deadband or the leg hasn’t reached the desired location, over time the error will accumulate enough to force movement. By not waiting for each motor to complete a movement, if the timer is set correctly the leg axis should move on to the next movement without having to stop. This will be fun to figure out how to implement, I wonder if setting up a spare cog on the master as a dedicated timer might work, have it set a flag or counter whenever the time interval is hit so the master knows its time to release the next set of movement commands.
Another suggestion was to calculate all the individual movement commands up front, store them in an array and pop them off the stack when the timer activates. If the master takes longer to complete the IK calculations than the time interval then this would be the way to go. Otherwise, if the P2 can perform all the calculations in less time than the timer, calculations can be done as needed. I need to run a time trial on the current calculation process, I believe I’ve seen some code somewhere that makes it easy to time your code, just have to find it again.
Looks like I have some more testing to do!
Hey Bob,
A pretty cool thing that I have going is a BASIC interpreter/editor right on the device (STM32H743 @480MHz). Code execution is similar to SPIN on P1 but I can simply edit/run without the download.
I am running a servo, no sweat. Both the PID and the position update happen in the same 1Khz loop. I stripped-out the irrelevant BASIC code so this is like pseudo-code:
"spdcmd" is encoder counts. Obviously, in this case, each increment of spdcmd = 1000 cts/sec which is just for clarity
spdcmd is where you put your target position divided by the loop-rate (1000 in this case)
Point being, put everything in the PID loop. Externally, you'll need to scale the commands and take care of velocity profiling etc.
But it doesn't get much better than this. 1KHz is more than adequate for pretty much any mechanical device and here it is running on an interpreted language.
I have much more going on than shown above but of my 1ms window, I'm a little under 250usec.
ArmMite H7 Basic interpreter closed-loop servo control (video).
Err = Following Error
Vel = Actual motor velocity (500 line encoder, 2000 quadrature counts/rev)
The other number is execution time. The P1 @80MHz should be similar with SPIN.
Craig
Correction: spdcmd cmdpos is where you put your target position divided by the loop-rate (1000 in this case)
I've started experimenting with using a timer to coordinate the 3 motors vs waiting for each motor to complete its movement.
I decided to first try implementing the clock on the master P2 instead of the individual P1 leg controllers. The reason behind this is the P2 is the source for all movement commands and I can more easily keep each leg in sync with the other legs by regulating the timing of when the P2 outputs a movement command. The P1 control software already processes an incoming P2 command even if the previous command has not completed. I'll try this method out first and see what kind of results I get.
All the coding is being done on the P2 master at this point. I set up a clock that gets incremented every 1 millisecond. Since I didn't know how fast the test routine ran, I timed the main loop in the program that is outputting data to the individual leg controllers. I found the loop was running about 142 usec, so I have lots of time available to play with.
Initial testing is with a single leg, telling it to move at a 45 degree angle for 200 millimeters. I set up a delay interval of 25 milliseconds just as a starting point. The leg visually seemed to be moving fine and then suddenly moved much faster to a final position. Review of the Debug logs shows the leg didn't actually start moving until the 6th movement command. Then the leg movement in all axis were running well behind the incoming command positions and the difference between commanded position and actual position kept getting larger. The final burst of movement I was seeing was the motors finally catching up the final position.
I increased the time interval to 1 second just to see how the motors responded, even there the motors were not keeping up with the commanded position. Plus, the final position the leg ended up in isn't where I expected.
So it looks like I made an conceptual error in the way I thought the leg movement should be calculated since the leg isn't getting to the expected position (drawing this out on paper I think I didn't account for leg position offsets). The fact the leg is just starting to move after the 5th or 6th command set means there is some backlash between the motor and the axis it is trying to move.
Got some more thinking to do!
Yup, PID doesn't like backlash. This is actually a pet-peeve of mine because the industry has kinda gone backwards. Most machinery, today, relies on the feedback device that is fixed to the motor shaft. I had a client call me out on what he believed to be a process problem (metal forming). He didn't question the control system because the display showed zero error. Well the motor shaft was where it had been commanded to be but it was totally oblivious to the 2mm of backlash in the drivetrain.
Many years ago, I came across some opposition from a particular mafia group of engineers within a large automotive company. They were totally opposed to the idea of me rebuilding and re-controlling their older machines. At the time, it seemed like every engineer was obsessed with six-sigma, continuous improvement, yada, yada and an old machine could never be as accurate/repeatable as a new machine. I knew what they were really concerned about which was the fact that they got a week of wining/dining/adult-venues on the West coast when they went to the manufacturer to sign-off on a new machine, prior to delivery.
So I threw down the gauntlet and guaranteed them that my rebuilds would out-perform any of their new equipment. Of course they laughed and gleefully accepted the challenge. What they didn't know was that I use dual-loop feedback. Motor feedback for axis stability and an auxiliary feedback device mounted on the actual load.
They went to great lengths to produce an apparatus involving their own bolt-on feedback devices connected to a data-acquisition PC (they didn't do this for new machines). The nominal spec was a repeatability of +/- 0.1mm (kinda sloppy by our standards) and so their external devices were good for +/- 0.05mm (the intention was to humiliate me). Well my "Sony Digi-Rulers" were good for +/- 0.01mm which was what we could hit, every single time
Their software was supposed to collect variable data and plot bell-curves etc. Imagine the look on their faces when they weren't able to record any error whatsoever. "How are we supposed to plot a bell-curve if we don't have any variable data?" Not my problem
I respectfully requested that we perform the same experiment on their newest machine but somehow they never got around to it...I got the contract
Having position on both the motor and load is a great way to really know what is going on. My system has the sensors on the load (the moving part of the leg) but nothing on the motor as the linear actuators are sealed units. I was looking into putting some type of optical reader inside the housing but there isn’t any real room (there is a 10 turn potentiometer inside the housing but it reads off the output shaft so not very helpful in this case.
I did a layout of the leg movement on paper and found a couple of areas I could improve the code and hopefully get better accuracy in the movement. I’m still considering options for dealing with backlash. Since I have an indication of when the load starts to move, I could send an initial movement command but not start the timer until feedback indicates the load is actually moving. This approach could get complicated since in some cases one or more leg axis movement command might be smaller than the deadband value.
Anyway, that’s half the fun of this project is finding a problem and then figuring out a solution that fits both my abilities and budget! First step is to code a example using the new ideas and see how that works out, hopefully not introducing too many new issues…
Methinks that the backlash between the actuator and the feedback has necessitated a low-ish proportional gain (Kp) to achieve stability, which could also explain the lag.
A useful trick is to apply an offset to the command, external to the PID loop. This offset will take-up the backlash, prior to the PID doing its thing (PID only responds to error so give it a push to minimize the error).
Am I interpreting this correctly: if a movement command is received that reverses motor direction (I’m assuming that continuous moves in the same direction have already taken up any backlash), outside of the PID initialize a a higher error value to get the motor starting up faster than the normal PID loop would allow? Or actually start the motor up and moving before the PID loop starts up?
Presumably, you have the standard:
And this is already calibrated for the optimum response/stability (limited, due to backlash)
Add an offset to provide a bias in the direction of intended motion
If the offset is set a little too high, the Kp+Kd+Ki will oppose it like it was a load disturbance so it won't run away
Using a new method for moving a leg in a straight line, I coded and tested the new setup. I also put in the ability to send the movement commands from the P2 to the P1 controllers at discrete intervals.
The method assumes that the user will input the direction of travel and the length of the stride of the leg. To keep things simpler, the height of the robot is fixed. Individual legs are assigned positions on an XY grid with the 0,0 point being the center of the robot and the legs located at 60º intervals in a circle around the grid. I assigned leg 1 to be located at 0º, so if the user puts in a travel direction of 45º, leg 1 will also move 45º, leg 2 is offset by 60º so its movement angle is 45º - 60º or 15º (based on the local leg 2 coordinates). Working with leg 1 to start (easier calculations and easier to visually see if something isn’t going right), I break up the distance into discrete intervals (using 5mm as a starting point). Dividing the total stride length by the interval gives the number of individual positions the leg will move. Next is determining the amount of X and Y movement needed for these discrete moves. If starting with a 45º angle with a length equal to the interval (5 mm = hypotenuse), the cordic comes back with the X and Y value change needed for each individual movement. The change in X and Y is added to the current X and Y position and the leg moves to the new position.
Previously the coding waited for each motor to finish its move before the next move command was sent. This results in a very jerky movement. As suggested by Mickster, I added a timer that controls when the movement command is released. This value is being determined experimentally (35 msec is looking good). Too long an interval and the leg move gets jerky, too short and the motors can’t keep up.
Current problem I’m also working through is backlash. During testing I reset the leg to a known position and then have it move to the desired location. What happens here is that at the start of the movement the motor starts turning but it’s about 250 msec before the leg sees any physical movement from the sensors. I tried a quick idea of adding an extra 250 msec to the first movement timer but that didn’t have the desired effect. Mickster recommended adding an offset to the PID loop in the PID code when a motor direction change is noted. I got back into the leg controller code and quickly realized earlier I had changed the PID quite a bit so that the femur and tibia were modified that it wasn’t really a PID anymore as the motors were just told to run one direction or another until they reached a desired position. The femur and tibia both use linear actuators which are fairly slow moving devices and the distances they would be asked to move would be fairly small so locking in the motor speed was the best solution. The coxa still uses the PID as it responds much more quickly, there I can experiment with an offset value. I was able to speed the motors up some. Next stage is uploading the change to the leg controllers and see how much of a difference that makes.
I would like to post some video of the leg movements but don’t know how to upload short video clips to this forum without going through YouTube. Is it possible to upload 10-15 second long video here?
Hi,
Just an idea. To get some more information it would be nice to know the actual speed of a motor. So did you consider to measure back electromotive force? This might be helpful because you actually have this speed sensor for each axis for free on the motor shaft.
Christof
Yeah, as noted by @ManAtWork, that 35msec rate would normally be jerky but it's clear that your overall loop gain is low (sluggish) in order to tolerate the lost motion (backlash).
Sounds like you're on the right track though except for going open-loop on the femur and tibia....
Once you get to grips with the discretized commands and offsets, I reckon that you could reinstate the PID.
Theoreticians tend to assume a perfect world and so pretty-much everything published, regarding PID is totally oblivious to reality. I have many hundreds of axes using a "totally unsuitable" (a PhD from Vickers-Rexroth) proportional valve. He was right.... horrendous dead-band but I got creative with the PID and arriving at +/- 1 encoder count became a breeze. I'd previously used a high-end servo-valve but even with 10 micron oil filters, they still failed (big bucks). These proportional valves, I could put sludge through them and they'd keep on trucking.
Now that he has a sample-time, he pretty-much has it; dp/dt=velocity.
Made the updates to the leg controller program to increase motor speeds under specific conditions. Initial testing shows the response time for the femur to reverse motor direction and take up backlash has been reduced to about 100 milliseconds vs 250 milliseconds. Testing shows that 35 millisecond loop time is where the motors start to jerk due to coming to a stop before the next command is received. 30 milliseconds seems to be the sweet spot with the current setup. I added a backlash timer that runs if any motor reverses direction (meaning there is backlash to take up) so the initial movement timer is increased to 50 milliseconds. This idea seems to be a work in progress.
I attached a copy of one of the debug logs and replicated some of first data it is giving me.
Cog0 Test 5x - move single leg in straight line using IK and timed movement
Cog0 Transmited command: $,1,828,75,900,1
Cog0 Leg 1, legXActual[n] = 0, legYActual[n] = 297, legZActual[n] = 610
Cog0 -------- coordinateConversion ----------
Cog0 Leg 1: moveAngle: 450
Cog0 Leg 1, bodystride = 200, legangle[n] = 450, resolution = 5, stepincrement = 40
Cog0 Start position for Leg: legXActual[n] = 0, legYActual[n] = 297, legZActual[n] = 610
Cog0 Step distance: x = 4, y = 4, z = 0
Cog0 Loop interval (msec): msecDelay = 35
Cog0 ---------- Start timed movement ----------
Cog0 Step: 1 Next Position Leg 1, legGaitX[n] = 4, legGaitY[n] = 301, legGaitZ[n] = 610
Cog0 Transmited command: $,1,827,76,892,1
Cog0 Leg 1, legXActual[n] = 4, legYActual[n] = 297, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 814 Tibia: 53 Coxa: 962
Cog0 Position error: femurAngle[n]-fangle = -13, tibiaAngle[n]-tangle = -23, cangle-coxaAngle[n] = -70
Cog0 ------- End Step -------
Cog0
Cog0 Step: 2 Next Position Leg 1, legGaitX[n] = 8, legGaitY[n] = 305, legGaitZ[n] = 610
Cog0 Transmited command: $,1,831,79,884,1
Cog0 Leg 1, legXActual[n] = 8, legYActual[n] = 300, legZActual[n] = 611
Cog0 Leg 1 angle values: Femur: 814 Tibia: 54 Coxa: 962
Cog0 Position error: femurAngle[n]-fangle = -17, tibiaAngle[n]-tangle = -25, cangle-coxaAngle[n] = -78
Cog0 ------- End Step -------
Cog0
Cog0 Step: 3 Next Position Leg 1, legGaitX[n] = 12, legGaitY[n] = 309, legGaitZ[n] = 610
Cog0 Transmited command: $,1,831,82,877,1
Cog0 Leg 1, legXActual[n] = 12, legYActual[n] = 303, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 816 Tibia: 54 Coxa: 962
Cog0 Position error: femurAngle[n]-fangle = -15, tibiaAngle[n]-tangle = -28, cangle-coxaAngle[n] = -85
Cog0 ------- End Step -------
Cog0
Cog0 Step: 4 Next Position Leg 1, legGaitX[n] = 16, legGaitY[n] = 313, legGaitZ[n] = 610
Cog0 Transmited command: $,1,835,88,870,1
Cog0 Leg 1, legXActual[n] = 16, legYActual[n] = 311, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 816 Tibia: 59 Coxa: 958
Cog0 Position error: femurAngle[n]-fangle = -19, tibiaAngle[n]-tangle = -29, cangle-coxaAngle[n] = -88
Cog0 ------- End Step -------
Cog0
Cog0 Step: 5 Next Position Leg 1, legGaitX[n] = 20, legGaitY[n] = 317, legGaitZ[n] = 610
Cog0 Transmited command: $,1,838,92,863,1
Cog0 Leg 1, legXActual[n] = 20, legYActual[n] = 315, legZActual[n] = 610
Cog0 Leg 1 angle values: Femur: 824 Tibia: 65 Coxa: 958
Cog0 Position error: femurAngle[n]-fangle = -14, tibiaAngle[n]-tangle = -27, cangle-coxaAngle[n] = -95
Cog0 ------- End Step -------
I attached the debug log from one of the test runs. The leg is moving at 45 degrees for 200mm. The individual step interval is 5mm so it takes 40 steps to travel that distance. The leg X,Y,Z position values are in mm. The Position Error shows how much difference there is between the requested angle and the actual angle of the leg during the step. I copied the first 5 steps where you can see the femur angle starts at 81.4 degrees (814 due to using integer math), the P2 wants the femur to move to 82.7 so there is a difference of 1.3 degrees. The femur stays at 814 until step 3 where it increases to 816 but stays there for the following step also (this is taking up backlash). By step 5 the femur is now moving but doesn't catch up. By step 32 the femur starts to fall behind progressively more each step. The tibia tends to follow the requested movement fairly well, only about a single degree difference. Coxa motor wanders faster and slower but maintains itself around 4 to 5 degrees behind the needed position. the very last 2 lines of the log show the final stopping point. all motors are only 2 degrees off from the final position.
I'm going to play with the backlash and timer interval values more along with the step interval distance, I've got some other ideas to try out based on some of the input from Mickster and some research I've uncovered.
Well, as there is so much backlash, he does not have it for a lot of time. Using the motor as a tachogenerator works instantly and surprisingly well with DC motors.
I ran into a problem during testing that had been very infrequent but lately happens almost every time I download a program update to the P2 controller. There are random leg movements from legs that aren't being sent a movement command. This seems to be specifically happening during the initial P2 setup routine and right after the ports for the leg have been initialized. It’s acting like the buffer has a valid command when nothing has been sent yet. Movement that isn't requested is a serious problem, luckily the robot is on a test stand with the legs not touching the ground so no damage so far. But I just had leg 6 move the coxa motor over far enough for it to interfere with leg 5 when it tried to move.
EDIT: I’ve been fixating in the idea that the P2 was randomly sending a valid movement command to the P1’s and I couldn’t find a source of the signal. I dug into the original HB-25 motor controller docs and found it only takes a single pulse to the controller in the range of 1-2 milliseconds to initiate a continuous PWM output. This corresponds with the movements have been for the full travel of the affected axis and only stop when they reach full travel. It seems likely that the HB-25 control driver is occasionally outputting a valid pulse while the P2 is initializing the communications ports.
Something else to look into today!
EDIT 2: Ever had a day when everything goes bad? Today was that day for me. I made what seemed to be fairly simple code changes, uploaded the changes and actually made the situation much worse! Trying to figure out why the changes were causing this much additional issues has been difficult and time consuming. I’ll roll back the updates tomorrow and try again.