Propeller Problem Persists

Discovery

I am using P2, 3, 4, and 5 to interface with the Analog To Digital Converter and it works properly.

I am using P16, 17,18, and 19 to generate the four phases to operate stepper motor #1 using "C" code HIGH() and LOW() commands.

I am using P20, 21, 22, and 23 to generate the four phases to operate stepper motor #2 using "C" code as above.

I am using P24, 25, 26, and 27 to generate the four phases to operate stepper motorr #3 using "C" code as above.

"C" code in the main program reads coordinates and drives motors #1 and #2 for X and Y axes.

"C" code in a COG runs motor #3 for the Z Axis and must run by its self.

If the main program runs motors #1 and #2 and does not start the COG there is no problem.

If the main program only starts the COG for motor #3 servo control there is no problem; however, if the code runs together (all three motors) one, two, or three phases generated by the COG stop running in either a high condition or a low condition. It occurs intermittently. All motors may run for several hours perfectly...then one, two, or three COG phases stop and the servo control fails. Or, the phases may fail in different conditions for several hours. Power up...power down does not correct the condition. Over and over. All components are mounted on a double sided PC board. All parts are cap by-passed.

Does any one know why this occurs and is there a work around to solve it?

Discovery

Peter Jakacki

Discovery wrote: »

I am using P2, 3, 4, and 5 to interface with the Analog To Digital Converter and it works properly.

I am using P16, 17,18, and 19 to generate the four phases to operate stepper motor #1 using "C" code HIGH() and LOW() commands.

I am using P20, 21, 22, and 23 to generate the four phases to operate stepper motor #2 using "C" code as above.

I am using P24, 25, 26, and 27 to generate the four phases to operate stepper motorr #3 using "C" code as above.

"C" code in the main program reads coordinates and drives motors #1 and #2 for X and Y axes.

"C" code in a COG runs motor #3 for the Z Axis and must run by its self.

If the main program runs motors #1 and #2 and does not start the COG there is no problem.

If the main program only starts the COG for motor #3 servo control there is no problem; however, if the code runs together (all three motors) one, two, or three phases generated by the COG stop running in either a high condition or a low condition. It occurs intermittently. All motors may run for several hours perfectly...then one, two, or three COG phases stop and the servo control fails. Or, the phases may fail in different conditions for several hours. Power up...power down does not correct the condition. Over and over. All components are mounted on a double sided PC board. All parts are cap by-passed.

Does any one know why this occurs and is there a work around to solve it?

Discovery

Yes, post your code and schematics etc. (Unless of course this is a guessing game)

Discovery

Company policy does not permit product code and schematics to be disseminated. It appeared that the Propeller specifications were perfect for the plasma cutter application. So, it was selected.

If a sheet of steel is placed in the machine and painstakingly leveled so that the sheet is about one millimeter below the cutting tip and the Z axis positioning servo Propeller COG is disabled, the plasma cutter cuts the steel perfectly to an accuracy of 0.0025 inches in the X and Y axes over an area of 6.5 feet long and 3.5 feet wide.

If the Z axis Propeller COG servo is enabled and the X/Y code is enabled, the X and Y work perfectly while the Z axis stepper quivers and does not servo.

If the X/Y COG code is disabled while Z axis code is enabled, the Z axis servo works perfectly tracking the distance voltage from the plasma cutter.

When both sections of code are enabled the Z axis servo fails with only 1, 2, or 3 waveforms driving the motors. The three stepper motor driver code is written into subroutines and identical except for the output pins. Twelve pins are used to output the stepper waveforms. These subroutines are called as needed by the main program "C" code or by the Z axis servo code. The Z axis COG servo calls only the Z axis stepper subroutine and the main program calls only the X and Y stepper subroutines when a cut is made. It seems to me that the Propeller should work perfectly in this application...using COGs for exactly this purpose.

I don't understand what is going on inside the Propeller to cause the problem...something to do with the registers possibly. At this point, I have only one workaround solution and that is to redesign the control board so that the Propeller outputs only the X and Y set of waveforms from the main "C" program while the COG reads the ADC and outputs only two control signals to a BS2e (Z axis UP and Z axis DOWN) that generates the four waveforms to control the Z axis stepper motor.

Sincerely,
Discovery

Mark_T

You need to eliminate the possibility of interference - sounds like a very elecrically noisy
environment there, so that's where I'd instinctively look first - how well are all the interfaces
protected from noise from motors and plasma arc? A 3.3V circuit is particularly sensitive
(24V is commonly used in industrial environments, for good reason).

ErNa

It is not difficult to understand, that a company is willing to use support from volunteers without feeding something back, but in this case the problem should be modified to cover a broad range of applications, so the forum members have a chance to propose solutions or even present solutions which are useful for the community.

ErNa

p.s. I have to dig my garden, if somebody is interested to improve his abilities in digging, he is welcomed, I can contribute the materials ;-)
Please bring with you a showel

Publison

If you can not release information to the public, you might try on of the Authorized Consultants:

http://www.parallax.com/support/authorized-consultants

You could have them sign a NDA.

Discovery

There is no problem with noise. The control board is on a test bench with no motors and no connection with the plasma machine. The same failure conditions exist on the test bench as in the system. It appears to me that the output registers of the Propeller are affected by internal Propeller manager code and crash with this application.

How would I make contact with a reputable consultant who knows the internal architecture of the Propeller?

Discovery

Publison

Discovery wrote: »

There is no problem with noise. The control board is on a test bench with no motors and no connection with the plasma machine. The same failure conditions exist on the test bench as in the system. It appears to me that the output registers of the Propeller are affected by internal Propeller manager code and crash with this application.

How would I make contact with a reputable consultant who knows the internal architecture of the Propeller?

Discovery

See Post #6. Contact information was included.

Parallax FAE's deal with quantity development so would probably not be available unless you are producing a product with significant volumes..

T Chap

There are a number of simple reasons that could exist, but it is only guesswork if no one can see the code or schematic. Maybe you need some simple test code that does nothing more than flash and LED or output some data. This way you can at least see if your Prop is functioning with simple code. If the Prop fails with simple code, then you have hardware issues. When you say power off and power on does not solve the problem, that is hard to narrow down to code or hardware since we don't know if your code is fool proof( enough stack space for all cogs?). But with simple test code that only flashes an LED, you can find out if hardware is affecting performance.

When random things happen, it is best to replace the Prop and resolder all connections and components. Your symptoms could be bad connections, bad traces on the PCB, etc. I think you are wasting time chasing the internal structure of the Prop.

xanadu

What about the software called ViewPort that could help debug this?

jazzed

Well, this could just as easily be a software problem. Anything from not enough stack space (as mentioned) to having multiple cog threads over-writing one or more variables.

Could it be a current surge when activating another moving axis?

pik33

There are some common errors with programming the Propeller.
For example the cnt register commonly used to make delays. It is 32 bit and then every ~50 seconds it changes from $FFFFFFFF to 0. If you forget about this you can make the code which will hang in random moments and then (1) hang endlessly (2) return to work after ~50 seconds and then hang in the next random moment (maybe after some hour). (3) do one loop every ~50 seconds
I experienced all these three options in my code

But then, without knowing anything, we can only look at the crystal sphere or make use of Tarot cards, or maybe pendulum to solve the problem.

And.. why C? Why not standard PASM/Spin?

And why sonething is done in the main program, and something other "in the cog"? This is not the Propeller philosophy of work. To get the most out of the Propeller you start the cogs to make your "hard work", then use the main program to "rule them all". So, run your motors and read your sensor using some cogs, and let your main program only to rule them: get measurements and set the coordinates.

ErNa

I am not experienced in C programming the propeller, but with spin I had the effect, running two cogs with identical program, only one instance of the data segments existed, so the two instanced (two motors) interfered. Every single motor went as intended, but when I run two motors, one stood still, the other made some mixed up movements

Duane Degn

pik33 wrote: »

There are some common errors with programming the Propeller.
For example the cnt register commonly used to make delays. It is 32 bit and then every ~50 seconds it changes from $FFFFFFFF to 0. If you forget about this you can make the code which will hang in random moments and then (1) hang endlessly (2) return to work after ~50 seconds and then hang in the next random moment (maybe after some hour). (3) do one loop every ~50 seconds
I experienced all these three options in my code

My favorite bug is to set I/O pin states (on the same pins) from more than one cog (this can be done but it's easy to make a mistake when more than one cog sets the I/O state of a particular pin).

Forgetting to increment long memory locations by four units is a close second.

Phil wrote a nice "Propeller Tips and Traps" article which includes other common bugs.

Discovery

I did not notice the link to consultants...thanks for pointing it out.

I have experience writing C programs...none with PASM or Spin.

The test I just ran was on the bench with no motors attached.

This C code is incredibly simple...simple subroutine calls.

ErNa Did you solve the problem?

One COG runs the main program, #2 runs a floating point processor, #3 runs the ADC, #4 runs the z axis servo with calls to a subroutine that steps through the four phase waveforms. After compiling, SimpleIDE shows plenty of spare memory. The stack size was doubled with no change in COG waveform problems.

All the PC traces were inspected under a high power magnifier with no detected problems. All solder joints were also inspected and verified as very good.

The important point to understand is that the servo COG waveforms fail every time the additional eight waveforms are generated in the main program.

There are no current surges in the bench setup.

The 12 outputs of the Propeller feed 7475 buffer drivers and the signals were monitored by a four channel scope. When the main program code is bypassed i.e. no signals output to motor connectors 1 and 2 the four phase signals out of the COG are perfect. When the main code is allowed to run with motor phase signals generated at the Propeller output pins, an additional eight signals...the eight signals to the motor connectors 1 and 2 are perfect but two of the four phases of the COG generated output signals were dead out of of the Propeller.

I decided not to use separate COGs to generate four phase waveforms for the X and Y drives since these signals must be phase locked. It is simple to do in the main program.

Discovery

Heater.

Discovery,

This is a hopeless question. We are presented with an unknown problem in a system comprised of unknown software and unknown hardware. A "work around" is requested. A work around for what? There is some suggestion in the threads title that this is a "Propeller problem" for some unknown reason.

Seems to me that you have created a system that is sufficiently complex that you know longer comprehend how it works. It may have subtle interactions between parts, both hardware and software, that you cannot fathom. It may suffer from external interference.

I'm not getting at you when saying this. I think many of us will have been in that situation before. Even very simple things can cause hours, days, weeks of frustration.

Whist I understand many companies like to keep their cards face down. I hope they have the money to back up such a policy. Either they let you hunt the problem down for potentially weeks or they pay up for a consultant to fix it. All of this might be more expensive than putting the code on github and letting us have fun trying to so spot issues.

Anyway, under the circumstances only general hints and advice can be offered.

Number zero: "Assume nothing". You can be looking at an "obviously correct" piece of code or hardware logic for hours only to find out some time later the problem is exactly there. Our brains are like that. Sometimes it's better to walk away for a day or two and come back when you have forgotten how it is supposed to work.

Number one "Divide and conquer". Trying to track down a problem in the system assembly is hell.

Does your software work correctly in test harnesses away from the actual hardware?

Do the individual parts of the software work correctly in their own units tests away from the overall program?

Have you tested those software components against all expected inputs and checked what they do for unexpected inputs?

Is the is software problem or a hardware problem?

Is this a fault in the hardware design or just a dodgy connection or other problem in the existing hardware build?

Is there some external interference that confuses the hardware?

We have one very big clue in what you have said:

"...may fail in different conditions for several hours. Power up...power down does not correct the condition. Over and over."

To my mind this rules out a software issue. After all a quick power cycle/reset should always get you back to the starting, presumably error free, point.

Unless that is, the software has some dependence on the current time. Do you have a calendar clock chip in there?

Or the software has some dependence on values it is writing to non-volatile store. Do you do that?

That points me to hardware. But where is the memory effect in your hardware? How does it get to be failed for several hours at a time and then recover? Do you have some big capacitors in there? Is there something external upsetting things via EMI or mains born interference that goes on and off on such time scales?

The last place I would be looking is to the Prop chip itself. It just seems a million times more likely to be your code or your hardware.

We are all going to be fascinated to discover the resolution to your problem.

Discovery

I forgot to mention that I changed the program to a new version that uses the z axis COG to only generate two simple signals. One signal is to raise the z axis and the other is to lower the z axis. These two signals were sent to a breadboarded BS2e that was coded to generate the four waveforms to turn the z axis motor. The test configuration works perfectly. The eight x and y axis waveform signals are correct and stable while the BS2e slews the z axis smoothly inacordance with the plasma cutter height simulated voltage.

This modified controller works perfectly in all bench tests.

Discovery

frank freedman

Discovery wrote: »

I forgot to mention that I changed the program to a new version that uses the z axis COG to only generate two simple signals. One signal is to raise the z axis and the other is to lower the z axis. These two signals were sent to a breadboarded BS2e that was coded to generate the four waveforms to turn the z axis motor. The test configuration works perfectly. The eight x and y axis waveform signals are correct and stable while the BS2e slews the z axis smoothly inacordance with the plasma cutter height simulated voltage.

This modified controller works perfectly in all bench tests.

Discovery

observations
1. 7475 is a 4-bit latch, not a buffer. Are the setup times etc right for the latches and does it have the fan-out to drive what you want? 74lsxxx lower drive capability compared to 74xxx line.
2. you moved the z from a cog to external processor, no more problem. sort of hardware problem, caused by software I would guess, z cog stepping on the I/O lines of another's lines (oddly in propeller, highs rule whereas in TTL logic lows will rule) or a software variable getting hammered because another routine thinks that space is the right one to write over. Try stuffing the binary into gear if possible or check out one of the debuggers found in some threads here.

Discovery

Heater,

An experiment just performed on the PC Board on the bench without any connections to stepper motors showed that all the eight waveforms were correct for turning stepper motors clockwise and counterclockwise. The signals were observed using a digital oscilloscope connected to the pins of the propeller.

The program takes a list of x and y coordinates that define the shape of the object to be plasma cut generates a number of pulses for stepping the motors. M number of steps for x and N for the number of y steps. The list of coordinates are floating point format such as 64.125". The program subtracts these floating point numbers in x and y to generate the delta x and delta y movements. The floating point delta numbers then generate the discrete values for M and N.

This is really simple stuff written in C.

The z axis servo running in a new COG reads an analog value and generates four phase signals to servo the cutter position to be within 1.000 volts dc and 1.005 v dc. When the voltage is between the two numbers no waveforms are generated. When the voltage is outside the range only two of the four phase wave forms are generated. This is the failure. The x and y portion of the program runs by itself just fine and the z axis COG program runs by itself just fine but the z axis COG program fails when the whole program is run.

The waveforms are clocked at about 12 kHz so there are 12 outputs that run at around 12 kHz while the floating point algorithm is calculating the next change and the z axis COG is measuring the ADC which is the same ADC as that on the Propeller Activity Board.

This is as simple a test as can be setup and the propeller will not output all twelve waveforms only ten.

I figure that something is happening to the output register control when 12 outputs are being clocked at 12 kHz.

The only solution I found is to use the z axis COG program to determine when the plasma cuter head height is in range or needs to be raised or lowered. It thus outputs an UP signal or a DOWN signal that is sent to a BS2e where the four waveforms are generated to turn the stepper motor clockwise or counterclockwise.

If a solution to the original problem is not found...I will fabricate the alternative or workaround solution since it works perfectly on the bench test. The original design is much simpler but if the propeller cannot output the signals correctly...the alternative will be implemented.

Sincerely,

Discovery

Cluso99

Are you running the same "C" program in two cogs? Maybe the variables are being shared and hence corrupted??? Just a thought without scouring the whole thread.

msrobots

The propeller can output more then 12 waveforms at a time on different pins. No question here.

Listen to Frank. You are somehow stepping on your own pins

I am quite sure it will be related to setting of dira on another cog, I guess when the two waveforms are missing the output pins are high (3.3v) because outputs are OR'ed together.

Quick test:

use different pins for missing waveforms, just for testing, say eeprom or serial or led or whatever you have onboard.

Double check all places in code where pins get set to output or input

Each cog has its own dira and outa register and they need to get set from within that cog

Then they get OR'ed together by the Propeller and send to the pins.

You said that all 3 are working alone but not together. So the xy motor routines are stepping on the pins of z running in a different cog

Enjoy!

Mike

Heater.

The problem is I don't see how it is possible that the program stepping on it's own variables, stacks or pins could account for this reported failure mode:t

"the phases may fail in different conditions for several hours. Power up...power down does not correct the condition. Over and over."

A quick reset/power cycle would recover from the corrupt data/pins issue but this seems to be a sticky fault.

Of course that could be a red herring which is just diverting attention from the real issue. Or there may be more than one issue in play here,

Discovery

The only variable that is shared is the Servo Voltage measured by the ADC using code in the z axis COG. Removing the sharing variable makes no difference to the problem. All the other variables are defined in the main program COG. No outputs are shared between COGs...they are twelve individual output waveforms.

I wonder if the propeller can handle the speed of 12 outputs clocked at 12 kHz and higher. The propeller outputs 10 output waveforms without any problem. Your question about running the same C program in both COGs makes little sense to me. The main COG program starts the z axis COG early in the code. The code in the z axis COG is very short...reading the ADC voltage, comparing the voltage to the 1.000 vdc and 1.005 vdc limits then generates the four 12 kHz waveforms for turning the z axis stepper motor.

All the code is written in C so the direction register is set automatically by C. All the 12 waveform outputs are output only and never switched to input. All inputs are on designated input only pins. No direction switching is involved. The z axis waveforms are set within the COG using C code HIGH(16) and LOW(16) for example. A variation was tried by calling a function from the z axis COG to a subroutine that set the output waveforms. The same failure occurred using both methods.

Sincerely,

Discovery

Heater.

Discovery,

I wonder if the propeller can handle the speed of 12 outputs clocked at 12 kHz and higher.

I'm very sure that it can. We have a four port serial object in OBEX that can operate far faster than that. One could easily run many instances of that, until one runs out of pins or COGS. No problem.

You keep persisting with the idea that this is somehow a fault in the Propeller chip. Personally I would be extremely surprised that you have stumbled across a bug in the Prop. The Prop has been pushed very hard in all kinds of directions for many years now.

It is possible however that you have a faulty chip. Have you tried replacing the Prop.

Also again, the situation you describe of "It occurs intermittently. All motors may run for several hours perfectly...then one, two, or three COG phases stop and the servo control fails. Or, the phases may fail in different conditions for several hours. Power up...power down does not correct the condition. Over and over." does not point to a bug in the Propeller design. How could it possibly "remember" to be failed across power cycles and resets for several hours?

Is that situation still true by the way?

Your conviction that the Prop is at fault is going to blind you to finding the actual problem in your hardware/software.

Discovery

No.

Discovery

Heater.

I presume that is a "no" to the long term failure issue. If so what changed?

Discovery

The NO is out of sequence. It refers to the question...Are you running the same program in two COGs? The answer to the question is no.

Discovery

Heater.

Ah, confusing. I'll hold off until we get back into synch.

Discovery

Heater,

My next experiment, when I return to the shop after the holidays, is to load a similar program into the Propeller Activity Board and observe all 12 outputs. Previously, I loaded the x and y drive code into the Propeller Activity Board and it ran without fault. I also ran the z axis code on the Propeller Activity Board and it ran the stepper motor via four HEXFETs and a potentiometer. The code worked perfectly. The complete code was not run on the Propeller Activity Board because there aren't enough input and output pins available.

The test code will perform ADC and floating point conversions and output all 12 waveforms.

I am not suggesting that there is a failure in the propeller architecture as some responses state but I am concerned that the SimpleIDE C code may be involved in somehow limiting the I/O function in some fashion.

Sincerely,

Discovery

frank freedman

Assuming the ADC cog is using a multi channel ADC, have you tested this function to see if 1, correct channel is selected 2, it is correct in its timing and generating the correct values and 3, these results are being stored in the correct shared location(s). Also if sharing a single location, are the values synchronized for the axis reading the value at a given point in time. Same may go for all axes reading their own location for position data, how does the axis know the value is valid. What would happen if the value jumps the track positionally. If the ADC is correct, what sort of test values have been fed into the FPU; does it handle boundary conditions well?

I wrote this response after re-reading some of the earlier parts of this thread, including the no connection testing, and thought the stepper issue may be only symptomatic of the underlying problem. Perhaps creating test plans and data to perform unit testing on the individual systems/subroutines may give more insight into the real problem. Maybe feed a sawtooth into the ADC and turn that around into a DAC and compare the waveforms, Feed data sequences expected into the FPU function an see if the result is what you expected. Individual unit testing may resolve the issue faster than jumping straight into full system testing.

Hope I an not to far out in the weeds,

Frank

Discovery

Frank,
Thank you for your comments. The replies to your questions follow.
1. Yes.
2. No on timing...yes on correct value.
3. No. As far as I know, there is no sharing. All inputs are single values on individual input pins as selected by the HIGH(xx) and LOW(xx) instructions. Same goes for the outputs...no sharing, each output waveform has its own designated output pin.

My code is in C and I don't know how to do the checking you suggested.

The bench test performed on the ADC used the Propeller Activity Board, four HEXFETs, a potentiometer to simulate the 0 to 8 volts cutter head position, and a stepper motor. The servo code in z axis COG reads channel 3 (0, 1, 2. 3) correctly. The positioning limits are 1.000 vdc on the low side and 1.050 vdc on the high side. When the code is running, only the z axis COG, ADC COG, and main program COG are running, the pot can be turned in the correct direction to move the stepper motor and increase the distance between the cutter tip and the work piece. The pot can be turned in the opposite direction and the stepper motor decreases the distance between the cutter head and the work piece, When the voltage input to the ADC is within the limits the output pulses cease and all is well. This function tests very well.

This system is completely deterministic based on the number of steps of the motors and the pitch of the precision lead-screw. In all tests of the system with only the x and y axes operating, the propeller code accurately moves the cutter head to any point on the cutting surface. There is no problem. here. The problem arises when both x and y axes are operating at the same time the z axis is operating. The z axis does not move very far since it servos the cutter tip to be 1.5 mm from the work piece.

Sincerely,

Discovery