It is a longish story, but it turns out there is a big-name electronic speed/phase controller that under a certain set of conditions tried to partially energize all three phases, At once. Or out of sequence. And the Prop caught it.
The working theory is the braking code was getting triggered in the controller. Because the controller thought it was *supposed* to be braking, it never threw an error. The braking code starts “soft”, so there was never a per-phase over-current condition, and the condition was short in duration (but was being retriggered multiple times per minute) so “hard” braking forces never happened. This caused not just thermal problems, but rotational harmonics. No wonder they were eating motors.
The data is just beautiful. The out-of-phase condition happens for a few dozen cycles, sync gets restored for a few cycles (during which time the controller punches the throttle pretty hard to get the motor back to its target speed), lather/rinse/repeat. Sometimes there would be many days of totally normal operation, and then the anomaly would repeat again. When the gremlins hit, we saw the motor case climb 8 degrees C in a minute or so. If the gremlins stayed for a while, the temperature just kept climbing.
Once the data was clear, my neighbor quietly threw a corporate-level fit. There are now two engineers inbound from Japan and a really high-end logging phase monitor coming in from Germany. A Baldor motor guru is already on-site and his black box has mostly validated what the Prop found.
It is amazing how much mischief a six pack of beer, a FLIP and $30 in spare parts can create!
That is some cool work you've done there. What all was being measured?
I gather the environment wasn't an issue for the prop then?
Thanks! The embarrassing part is my code was/is a mess and not even close to being done. Site access was problematic as was weather, and then part of the plant went into a shutdown/turnaround phase making my engineer-friends time very scarce.
We monitored four different temps (nitrogen feed, case, 2 bearings) and “ambient” by using the DS3231 internal registers. Also, three phases of the ac line for voltage and current (via CT’s). Plans were in the works for rpm and to interface to several RTD’s embedded in the motor windings, but we never got that far.
This project, by any reasonable standard, is just a total kludge. No specs. No requirements. No budget. Not much time. Just a “hey! Lets try this...” and off we went. We took it from the bench to the field with literally a days notice when we saw a weather window. Some things we couldnt really test, so we just winged it. The current monitors were wonky and my scaling for the AC voltage measurements were off, but even so it was enough to get some useful data.
And yeah... the environment there is just brutal. But discovering that “cold” nitrogen feed ultimately made it possible.
There is an update to this story that has me sort of chuckling.
The official management response now is “yes. Wonderful. You found a problem. Thank you. Please dont let the door smack you in the posterior as you leave”. We sort of expected that. Its pretty much the standard reaponse when middle management realizes they cant hide something any more and they didnt invent the answer.
The second thing they did however, was *not* expected. We can not prove this policy change was triggered by our gadget, but the guy who drafted and signed it is the same guy who lost a bunch of skin due to a certain rogue Propeller project. The new policy is that “any field device that has not been formally vetted by IT and Site Management may not export or transfer data offsite via a hard disk, USB memory devices, tape, internet connection or wireless data transfer”. In other words, they want to control the access to data to prevent any future “analysis”. Not that I care. I’ve had my fun.
But then Herr Neighbor calls me. And of course we just had to have a beer to have a good laugh about this. And as we sat there on the porch, and as I read that memo for the second time, I suddenly realized that it did not exclude FLOPPIES.
I just looked at the standard 3-1/2 floppy interface. Its trivial. A Prop1 could totally do this. So yeah... there goes another weekend!
The "Kill the messenger" mentality is very common in large businesses when it makes someone high up "look bad".
Even if it was not their fault, and is for the good of the company.
I have known several people who got fired for not "being a team player" when they have done such things.
It is a big source of ongoing problems in a business, because no one wants to tell the boss that something he did is causing it.
I watched the "Snowden" movie a couple weeks back and found myself actually surprised that he had run into the same within the CIA just by reporting on network weaknesses when conducting intrusion testing.
Love your story and the FLiP side of it. Surprised (or Not) that management took the position that they did. I worked for a major international defense company a long time ago. We were developing a robotic machine gun system - No one checked the muzzle velocity and elevation angles to see if the gun met the range requirements!!! A junior engineer did some back-of-the-envelope calculations and found the rounds would be hitting sand 100 percent of the time LOL Everyone forgot something called *gravity* !!!
Temperature problem is apparently solved. The next item of interest is the massive electromagnetic pulse generated by starting the 3-phase motor. If the shielding is not adequate, the P1 program could be corrupted.
Comments
It is a longish story, but it turns out there is a big-name electronic speed/phase controller that under a certain set of conditions tried to partially energize all three phases, At once. Or out of sequence. And the Prop caught it.
The working theory is the braking code was getting triggered in the controller. Because the controller thought it was *supposed* to be braking, it never threw an error. The braking code starts “soft”, so there was never a per-phase over-current condition, and the condition was short in duration (but was being retriggered multiple times per minute) so “hard” braking forces never happened. This caused not just thermal problems, but rotational harmonics. No wonder they were eating motors.
The data is just beautiful. The out-of-phase condition happens for a few dozen cycles, sync gets restored for a few cycles (during which time the controller punches the throttle pretty hard to get the motor back to its target speed), lather/rinse/repeat. Sometimes there would be many days of totally normal operation, and then the anomaly would repeat again. When the gremlins hit, we saw the motor case climb 8 degrees C in a minute or so. If the gremlins stayed for a while, the temperature just kept climbing.
Once the data was clear, my neighbor quietly threw a corporate-level fit. There are now two engineers inbound from Japan and a really high-end logging phase monitor coming in from Germany. A Baldor motor guru is already on-site and his black box has mostly validated what the Prop found.
It is amazing how much mischief a six pack of beer, a FLIP and $30 in spare parts can create!
I gather the environment wasn't an issue for the prop then?
Thanks! The embarrassing part is my code was/is a mess and not even close to being done. Site access was problematic as was weather, and then part of the plant went into a shutdown/turnaround phase making my engineer-friends time very scarce.
We monitored four different temps (nitrogen feed, case, 2 bearings) and “ambient” by using the DS3231 internal registers. Also, three phases of the ac line for voltage and current (via CT’s). Plans were in the works for rpm and to interface to several RTD’s embedded in the motor windings, but we never got that far.
This project, by any reasonable standard, is just a total kludge. No specs. No requirements. No budget. Not much time. Just a “hey! Lets try this...” and off we went. We took it from the bench to the field with literally a days notice when we saw a weather window. Some things we couldnt really test, so we just winged it. The current monitors were wonky and my scaling for the AC voltage measurements were off, but even so it was enough to get some useful data.
And yeah... the environment there is just brutal. But discovering that “cold” nitrogen feed ultimately made it possible.
Nice job on the kludge and the fact that a big-buck 'bit of kit' verified your results is a nice touch.
-Phil
The official management response now is “yes. Wonderful. You found a problem. Thank you. Please dont let the door smack you in the posterior as you leave”. We sort of expected that. Its pretty much the standard reaponse when middle management realizes they cant hide something any more and they didnt invent the answer.
The second thing they did however, was *not* expected. We can not prove this policy change was triggered by our gadget, but the guy who drafted and signed it is the same guy who lost a bunch of skin due to a certain rogue Propeller project. The new policy is that “any field device that has not been formally vetted by IT and Site Management may not export or transfer data offsite via a hard disk, USB memory devices, tape, internet connection or wireless data transfer”. In other words, they want to control the access to data to prevent any future “analysis”. Not that I care. I’ve had my fun.
But then Herr Neighbor calls me. And of course we just had to have a beer to have a good laugh about this. And as we sat there on the porch, and as I read that memo for the second time, I suddenly realized that it did not exclude FLOPPIES.
I just looked at the standard 3-1/2 floppy interface. Its trivial. A Prop1 could totally do this. So yeah... there goes another weekend!
Just stirrin' the pot......
Even if it was not their fault, and is for the good of the company.
I have known several people who got fired for not "being a team player" when they have done such things.
It is a big source of ongoing problems in a business, because no one wants to tell the boss that something he did is causing it.
Bean
It doesn't mention a bunch of older technologies that are still viable.
Was that thanks but no thanks message only verbal or do you have something in writing?
Discovery