Project- 1st phase (informative)
Rforbes
Posts: 281
So, I've asked a lotta questions in these parts, and all of you have been really great with your responses, your patience and your knowledge.
I'm starting this thread to outline a project we're working on, so that if any of you have an interest in a specific aspect of what we're setting out to achieve, you can ask questions or provide commentary here.
The project isn't quite as complex as a manned shuttle to Mars
but we've implemented a few nifty features that help achieve our goal quite a bit. And of course, there is always room for improvement and modifications.
Here goes!
Background:
Our facility has a water plant utility and distribution system with the following relevant operational requirements and limitations.
The plant consists of 3 reservoirs (concrete tanks) of varying capacities, built and placed into service over several decades. These reservoirs are located not-so-strategically such that our distribution system is a multiple loop pipeline with several interconnects and isolation valves between each tank and the service lines for the folks we provide water for.
Over the years, lines have been placed into and taken out of service, to the point where our print box is overflowing with outdated drawings and our "as built" set of drawings consists of a D size "?" and coffee stains... and not much else.
Additionally, the 2nd and 3rd tanks were situated exactly where they were supposed to be, except... well, they weren't. The location is correct, but the elevation of the overflows for these two tanks are approximately 5 feet above the elevation of the overflow for the first tank. Since these two tanks are the largest of the three (220,000 and 190,000 gallons) we sorta kinda really, really need this lost capacity. Of course, needing it versus having it is a different story altogether. Suffice to say, our system *should* have a storage capacity of around 530,000 gallons but we can only hold about 73% of that.
The area we serve is small, but fluctuates in population from around 600 people to 6000 people during various parts of the year. During maximum population density times the "people" are mostly youth and young adult age. If you have children, you might quickly guess that instead of averaging 27 gallons of water use per person per day, we can easily average 98 or more gallons per person per day. Not counting extra showers for food fights, mud pit contests, and who knows what else. The point is- we go from very little usage to overwhelming demand in the blink of an eye.
Our water system is solely fed via wells. We have wells all over the place... and we're in the mountains. Goats die of thirst trying to climb up to our wells (ok not really, but they sweat a lot!) Some of our wells are not in pristine condition. The rest are just plain junk, but it's all we've got- unless we fire up our transfer plant and suck water out of our neighbor towns' water system (which is difficult, time consuming, labor intensive, expensive and and just plain un-fun.)
The controls we currently use for our wells are they latest and greatest 1970's era electro-mechanical timers you can buy for less than 50 bucks each. That's it. Nothing else. No flow meters, no pressure transducers, no trained monkeys, nothing. We set a timer for each well for up to 12 hours per day and let it pump into our distribution system, thereby filling our tanks. If we pump too much, we overflow and lose it. If we don't pump enough the people we serve get angry and lose it.
Of course, we chlorinate and phosphonate as required, and we check the level of our #1 tank daily to ensure we're not running too low on water, but overall we have several instances of wash-out (getting too low in the tanks and pulling the silt out of the tanks- into the system and into every washing machine connecting to the the system) and overflow each month. We lose approximately 15% of what we pump into the tanks to overflow and 20% of the time we have silt in the distribution system from wash-out. I'd love to tell you exactly why this happens so often but it's not appropriate for this forum.
To top it off, we not only lose water to overflow but also the Sodium Hypochlorite and Polyphosphate chemicals. And all the energy used to drive the wells. And the run-time hours that just ticked off for no good reason. On and on and on... you get the idea.
But lastly, don't forget the age of our system. We're approaching 100 years old on some of our lines. We have more distribution and service line blowouts than a wet willy lawn sprinkler. And since we're gravity fed, the 4 " and 6" lines at the bottom of the system (around 85 psi) make a really fun geyser to play in when they cut loose.
In all, here's what we're working with:
Excessive lost capacity due to construction errors
Aging wells pumping less than 70% of original flow rates
Crude controls for adjusting supply to the tanks
Manual adjustment of treatment chemicals
Excessive kwh consumption in well production
Excessive lost product due to a combination of factors
Very poor operational efficiency and reliability due to all of the above
Shoe string budget
Parallax and Digi International are coming to the rescue here.
That's it for now. I'll update this soon.
Robert
I'm starting this thread to outline a project we're working on, so that if any of you have an interest in a specific aspect of what we're setting out to achieve, you can ask questions or provide commentary here.
The project isn't quite as complex as a manned shuttle to Mars
but we've implemented a few nifty features that help achieve our goal quite a bit. And of course, there is always room for improvement and modifications.Here goes!
Background:
Our facility has a water plant utility and distribution system with the following relevant operational requirements and limitations.
The plant consists of 3 reservoirs (concrete tanks) of varying capacities, built and placed into service over several decades. These reservoirs are located not-so-strategically such that our distribution system is a multiple loop pipeline with several interconnects and isolation valves between each tank and the service lines for the folks we provide water for.
Over the years, lines have been placed into and taken out of service, to the point where our print box is overflowing with outdated drawings and our "as built" set of drawings consists of a D size "?" and coffee stains... and not much else.
Additionally, the 2nd and 3rd tanks were situated exactly where they were supposed to be, except... well, they weren't. The location is correct, but the elevation of the overflows for these two tanks are approximately 5 feet above the elevation of the overflow for the first tank. Since these two tanks are the largest of the three (220,000 and 190,000 gallons) we sorta kinda really, really need this lost capacity. Of course, needing it versus having it is a different story altogether. Suffice to say, our system *should* have a storage capacity of around 530,000 gallons but we can only hold about 73% of that.
The area we serve is small, but fluctuates in population from around 600 people to 6000 people during various parts of the year. During maximum population density times the "people" are mostly youth and young adult age. If you have children, you might quickly guess that instead of averaging 27 gallons of water use per person per day, we can easily average 98 or more gallons per person per day. Not counting extra showers for food fights, mud pit contests, and who knows what else. The point is- we go from very little usage to overwhelming demand in the blink of an eye.
Our water system is solely fed via wells. We have wells all over the place... and we're in the mountains. Goats die of thirst trying to climb up to our wells (ok not really, but they sweat a lot!) Some of our wells are not in pristine condition. The rest are just plain junk, but it's all we've got- unless we fire up our transfer plant and suck water out of our neighbor towns' water system (which is difficult, time consuming, labor intensive, expensive and and just plain un-fun.)
The controls we currently use for our wells are they latest and greatest 1970's era electro-mechanical timers you can buy for less than 50 bucks each. That's it. Nothing else. No flow meters, no pressure transducers, no trained monkeys, nothing. We set a timer for each well for up to 12 hours per day and let it pump into our distribution system, thereby filling our tanks. If we pump too much, we overflow and lose it. If we don't pump enough the people we serve get angry and lose it.
Of course, we chlorinate and phosphonate as required, and we check the level of our #1 tank daily to ensure we're not running too low on water, but overall we have several instances of wash-out (getting too low in the tanks and pulling the silt out of the tanks- into the system and into every washing machine connecting to the the system) and overflow each month. We lose approximately 15% of what we pump into the tanks to overflow and 20% of the time we have silt in the distribution system from wash-out. I'd love to tell you exactly why this happens so often but it's not appropriate for this forum.
To top it off, we not only lose water to overflow but also the Sodium Hypochlorite and Polyphosphate chemicals. And all the energy used to drive the wells. And the run-time hours that just ticked off for no good reason. On and on and on... you get the idea.
But lastly, don't forget the age of our system. We're approaching 100 years old on some of our lines. We have more distribution and service line blowouts than a wet willy lawn sprinkler. And since we're gravity fed, the 4 " and 6" lines at the bottom of the system (around 85 psi) make a really fun geyser to play in when they cut loose.
In all, here's what we're working with:
Excessive lost capacity due to construction errors
Aging wells pumping less than 70% of original flow rates
Crude controls for adjusting supply to the tanks
Manual adjustment of treatment chemicals
Excessive kwh consumption in well production
Excessive lost product due to a combination of factors
Very poor operational efficiency and reliability due to all of the above
Shoe string budget
Parallax and Digi International are coming to the rescue here.
That's it for now. I'll update this soon.
Robert
Comments
Now that we have a good background of what we're up against, it's time to develop a scope of work that "solve" our problems or at least mitigate the impact of them.
In a nut shell, we need to implement a solution that has the following characteristics:
-Minimizes time spent manually setting well timers. (Since wells are scattered throughout the woods, this currently takes up to 2 labor hours per day.)
-Prevents overflow from over-pumping the wells.
-Minimizes power usage for the wells.
-Prevents wash-out from the tanks when possible (The "dirty water" from this has wreaked havoc on our boilers, water heaters, fire suppression systems, etc)
-Minimizes water treatment chemical losses.
-Eliminates over-operation of the wells (By code, we cannot run any well more than 12 hours per day. It's bad for the water table.)
-Regains lost storage capacity.
-Allows remote monitoring of well run status and tank level.
-Requires no additional infra-structure to support. (We have power ran to each well, and that's all we're gonna get. No ethernet, phone, cable, satellite, etc will be coming.)
-Ease of use, operation and maintenance (We are a skeleton crew. There will be no additional manpower brought in to help and no contractors to perform service calls. We're on our own with this, solely in-house for everything.)
Since we're on a shoestring budget, it's important to not only keep costs down, but ensure every dollar spent is spent wisely. We have to make it happen with a ridiculously low budget since we're a not-for-profit organization. (There is no honey pot of cash waiting to be spent.)
Fortunately, the one thing we have is plenty of is time. There is no rush, other than the common-sense approach that "it ain't gonna fix itself" and knowing that more time to solve this issue means more wasted resources until it's fixed.
Now we have a broad based scope of work, so it's time for the fun to start by getting some details in place.
If you've read this far, you might be wondering why this isn't put in the "Projects" forum. Well, it probably should be... but since the Spinneret is key to the success of this project, and since Mike G and others who visit this particular section have contributed so much by answering questions for me, I figured this is a better place to lay this all out. Credit where credit is due, etc. etc...
On with the hardware selection:
We had meetings and discussions, and more meetings to hold more discussions and discussions about holding meetings and all the other good stuff that comes along with trying to kick off a project that involves several departments within an organization. In the end, we made a decision to use micro-controllers and the like to build a custom system from the ground up, instead of using PLC's and other fancy doo-dads which are exceptionally well suited to this type of project but also a lot more expensive and more difficult to support in-house. The ultimate reasons driving our decision are fairly straight forward: Cost per system component and the ability to modify, expand and improve the overall system operation without incurring large financial costs at any given point along the way.
After doing a bit of research and putting in a fair amount of google time, we decided on the following skeleton for the monitoring/control system.
1 Spinneret
10 Quickstarts
10 Xbee's (XSC, S3B version)
10 adapter pcb
1 ultrasonic sensor
10 TC74 temperature sensors
power supplies, antennae, enclosures, cabling, SSR's, other relays, and all the other good stuff that goes into putting together a reliable system.
We will handle our lost storage capacity problem by installing a 2 way altitude valve at our tank #1. This valve will shut the in-fluent to the tank when the system static pressure is above X, thereby allow the other two tanks to fill to full capacity Z. Once those two tanks lower to Z-Y level, the system pressure will drop accordingly and the altitude valve at tank #1 will open, allowing flow once again. This is quite an interesting mini-project unto itself, and will work hand-in-hand with this monitoring/control system project.
We have a ton of support from our IT department at corporate H.Q. and all the server side resources we can shake a stick at.
We decided to use the Spinneret as a client on our intranet and it's sole purpose is to gather data from the rest of the hardware and send it to a database. The database allows us to use a collection of intranet web pages that our water plant operators can view to remotely monitor the condition of the entire water system. Appropriate firewalls and other security measures are in place to keep unauthorized people from tampering with the system. The Spinneret makes good use of the micro SD card and RTC, both of which are critical for increasing the reliability and useability of this system.
The "Base" unit consists of the Spinneret, an adapter board, a quickstart and an Xbee.
The spinneret uses serial comms to work with the quickstart.
The quickstart uses serial comms to work with the Xbee.
The Xbee works with the Xbee's of 7 "Remote" units.
The Remote unit consists of a quickstart, an adapter board and an Xbee as well as other items appropriate to the particular task of the remote unit in question.
One of these remote units uses our ultrasonic sensor at one of our tanks to measure the water level.
The rest of the remotes are placed at our well houses, where they are connected to H/O/A switches, current sensing switches, ssr's and the like.
The remaining Xbee modules use an adapter board and power supply and are placed to act as system-wide repeaters. Our terrain is such that it's impossible to communicate via RF from any given point to every other point in the system, so there's no way to get around using repeaters.
More soon.
We have several requirements that have to be met in order for the state to approve the use of this system:
-Wells shall not be operated more than 12 hours per day. This time limit exists in order to ensure the water table isnt depleted too frequently. If we pump all the water out, were left with an empty cavity this cavity will be quickly filled back in by one of two things- more water, or earth from a cave-in.
-Wells shall not continue to operate in the event of extended communications failures with the controller. If the remote node doesnt correctly communicate with the base unit for a certain period of time, we have to ensure the well will automatically shut off. The only way we can allow the well to run is via the timers we currently use or by verification of the data the remote receives. Data errors and/or timeout failures will cause the well to shut off until valid data is received within the allotted time frame.
-Wells must be capable of operating manually or via timers when the controller or its subsystems or not operational. Since we have a fiduciary duty to provide customers with clean, safe drinking water in adequate supply we have to ensure we can run the wells without the control system. People gotta bathe!
-All applicable codes must be adhered to in the installation and operation of the control system. We cant just throw this together with breadboards and 9v batteries and let it run.
-Data reporting to the state authority must conform to current regulatory requirements. If we automate our reporting, we have to do so in such a way that our data is presented per the regulations. Consumption, production, runtimes, chlorine and phosphate levels, etc. all must be included and formatted correctly for the state to accept it. Of course, we dont have to automate this- filling in a spreadsheet like we currently do is fine. But in the end, well have this automated.
-Any and all upgrades must be approved by the well inspector as a documentable improvement. In other words, the inspector has to approve what were doing, and he or she must be able to document the hows and whys of it in justifying that it is indeed an improvement and not something that will hurt our system, the environment or our consumers.
-Security concerns must be met with the inspectors approval. Surprisingly, this is not so difficult to achieve. Regardless of what we do, we are considered a manned station in that we have personnel who are in responsible charge 24/7. We dont have someone looking at the whole system around the clock, but we have someone available to do so whenever its needed. This alleviates our need for heavy cryptographic communications and such. Even so, were ensuring that it will be extremely difficult for someone hack our system and do anything to it. Worst case scenario is that a well gets turned on or off when its not supposed to, at which point our personnel respond accordingly.
-Other requirements exist that are irrelevant for the intent of this forum post. If you're a water utility professional, send me a PM and I'll be happy to fill you in!
We now have a decent understanding of our issues, desired solutions, regulatory requirements, limitations and budgetary constraints. I realize I havent written out every single detail here but if I were to import the entire project folder the forum might implode on itself and turn into a working copy of Ataris PONG game. Theres just too much to cover in depth, so this is more of a Heres how we did it, thanks to Parallax, Digi and the folks on the forum.
We know what we need, but we want some extra goodies to go along with it. Thats the beauty of using our selected hardware. Were not limited to a specific function and we are free to add or remove things as wed like.
What follows is a working description of the system as a whole.
Base unit:
Upon system start up, the following occurs:
The spinneret will be logging data to a database that has a table for the base unit, each remote unit, an error table, messaging/notes table and an alarm table. The spinneret inserts records into each table every minute.
The spinneret gets hold of our in-house NTP server and updates the RTC. If the server is down, well automatically check up to 6 other servers. If we fail to update the RTC we keep the current values and log an alarm in our database. At the beginning of each day, we re-sync with the NTP server.
The spinneret then communicates with the quickstart and we continually loop, sending and receiving various chunks of data back and forth. We send the quickstart the current date and time and in return we get data from the quickstart.
Every minute, the spinneret uploads our data to the appropriate database table. If the upload fails for any reason (connection failure, no response, etc) it stores this data in a file on the micro sd card.
Once its finished uploading the live data, it then looks at the sd card and if there is data in the files, it will attempt to upload it from the file. If it succeeds, the data is removed from the sd card file. If not, itll try again the next time around. Data is stored in the files on the sd card as a FIFO buffer scheme. If the database detects an error in the data, the file is deleted (this has never happened, but its there to keep from continually trying to send bad data.)
As it stands, the network connection can be down for about 48 hours and as long as it reconnects before then we wont lose any data whatsoever. It will take a bit of time to upload all of the data from the sd card, but it *will* get there. After 48 hours, well lose 1 minute of data for every 6 additional hours of data on the card. This is a bit convoluted sounding but the overall scenario is such that weve minimized writes to the sd card, ensured that we dont skip a beat communicating with the remotes and dont have any time lag with regards to updating our run time values with the timers.
So far, we have had a few net outages and occasionally we dont get a response from our server. When we dont get a response, we consider the upload attempt to have failed. So we log the data to the sd card because we havent proven that the data was accepted into the database by getting an appropriate response. However, the data has always been accepted on the first try its just that we timed out before getting a response. Eventually, the data gets uploaded and we receive a response, so it works out well enough in the end. The only issue is that we get a few duplicate records in the database now and again, so we handle this with a CRON job, which runs every so often and deletes duplicate records. It's better to "assume" we didn't successfully upload data and get a duplicate record than to assume the opposite.
The quickstart on the base unit uses the date and time from the spinneret to accumulate run times for each i/o point on each remote node. The timers are reset each day, and serve to automatically shut off a given well if 12 hours run time has been reached in a single day. Throughout the day, the operator can see the accumulated run time. The resolution of the timers is about 6 seconds, due to the remote/base relationship and how the base polls each remote in turn.
When the base successfully communicates with a remote, we take our NTP time and add 180 to it. This is our auto shut-off time. If we dont talk to the remote again before that time, we set the remote to a disabled state. The remote has the same watchdog feature. If it doesnt talk with the base in time, it automatically disables itself.
Every minute, our latest runtime values are stored to eeprom on the quickstart. Any time we power up the base unit, we look at the current date and ensure we fetch the latest run time hours value from eeprom (or dont if we power up on a new day.) So, we basically record our values to a certain location in eeprom based on the day of the month and read it from the location as needed.
The Remotes
The remote units provide i/o points for each well, the chemical pumps and also the temperature sensors and ultrasonic sensor for tank level measurement.
The temperature sensor simply tells us enclosure temperature. This helps us ensure the heaters are working during the winter.
The ultrasonic sensor is a MaxSonar model and works beautifully. The data is sent to the base, where we use two simple hi/low/deadband limit models to activate io points on the various remotes for well operation. We use two sets of limits to help us conserve a bit of energy. One set is for peak hours and the other is for off-peak hours. Those kWh add up in a hurry and the operating principle is that well allow our tank level to get a little lower during peak hours and then pick up the slack during off peak hours to catch back up.
Speaking of peak/off-peak hours, we incorporate DST automatically on the spinneret RTC to avoid problems with time changes.
Since weve incorporated an altitude valve into our system, we use the ultrasonic sensor on the tank closest to our facilities building weve included an appropriate OOL method to handle errors from the sensor which in effect tells the base to shut off all remotes until we get it fixed. So, if we get 0 from the sensor it means weve either blown out a tank and lost all our water or the sensor is having issues. Either way, we dont want to run the wells via the control system without correcting the problem first.
Each remote also incorporates a Hand/Off/Auto switch and current sensing switches.
The H/O/A switch simply takes a particular output and disables/enables it. The remote receives command bits from the base, but sends status bits back to the base. So, when we place a remote output point in any position, we see it back at the base. The H/O/A switch also provides a hard electrical isolation from the well starter. Since the main electrical circuit controlling the well starter is wired in parallel with the old mechanical timers, we can still run with the timers when the switch is in HAND. When its OFF, its a complete isolation and lockout. In Auto, it bypasses the mechanical timer and runs according to the base commands.
The current sensing switches are a CT with a transistor output. The CT is a split core type, so its easy to install around the primary conductor to the well pump. Its used as a proving switch, where it senses current flowing through the pump motor, and provides an input to a pin on the propeller.
Debounce timers are incorporated for all i/o, as are staging timers for each well and chemical pump. The staging timers are important so that when the system tells the wells to start, they dont all start at once. The stage timer is user-settable, and weve set this to 5 seconds. (This minimizes peak load demand at any given instant and minimizes production manifold pressure surges.)
The Xbees
Were using AT mode for all our data here, nothing fancy. There isnt a lot of data to be transferred and incorporating a checksum and such wasnt difficult. We track the number of good packets, bad packets, checksum failures and timeout failures and the spinneret uploads this data to a table along with the other stuff. We have occasional time-outs, but overall the good packet count is over 98%. To minimize false negatives, we make the base and remote units wait for 3 consecutive communications failures (for any reason) before initiating an automatic disable of the remote node. This way, were assured that 1 bad packet every couple hours or even days doesnt cause a stop/restart the well pump unnecessarily.
To date, we have over 1 million records of data and the system is going strong, collecting data from 4 square miles of populated, forested mountain terrain. We can run up to 10 base units and 100 remotes without a hitch, and 10 times that with a little bit of code modification.
Were planning on expanding the system to include water temperature monitoring and room temperature monitoring for our facility, which has approximately 100 water heaters and boilers, and 800+ rooms. Well slow down our monitoring frequency for this portion of the system, probably 15 to 30 minute intervals.
Well also be installing units on all of our water meters, because we currently have to take physical readings to determine consumption/billing numbers. Even though were a not-for-profit organization, we do have to charge for water but its the cheapest in the state!
That's pretty much it for now. It's been a GREAT challenge and wouldn't have been successful without help from you all. I'm enjoying learning more about the propeller and everything that goes with it. (It's quite different from ControlLogix PLC's and GE Fanuc units!)
Please let me know if you'd like to know anything more specific about what we've done, why we've done it or how. I'll do my best to answer up but I'm sure I'll be posting lots more questions in the near future!
Robert