Phantom Channels

[OurMikeyRocks]

I have a network with 19 controllers - one DIO, two DCs, one 1602, and 15 CTB16PCs that I built from kits. I added two of the CTB16PC controllers this season to support a spiral wrap in my megatree.

Sometimes channel 1 and maybe channel 2 on one of the controllers just comes on without being sequenced. When the controller is being address the channel work as expected, but seem to perform this phanton turn-on randomly. I can't see a pattern to when this might be.

So, here's what I have done so far. I reset both new controllers. I reloaded the 4.32 firmware. I reseated all of the Cat-5 connections. There are both controllers before the problem controller in the daisy chain and after. Everything else is working as expected.

I have seen this happen on different songs and I have checked a couple for accidental commands that might be causing this - didn't find anything. I even went into the XML files to look for odd things and didn't find anything there.

Anyone got any ideas?

[jimswinder]

Have you run the "Verifier" to check for possible duplicate channels?

There were times I was 100% positive I did not have duplicate channels...I was wrong...

Edited December 17, 2012 by jimswinder

[George Simmons]

I had an issue like that. I've seen a lot of people this year with that type of problem. In almost all cases it was a bad comm cable.

You say you've "reseated" the cables. Try replacing the one that feeds the controller where strange things are happening.

As Jim recommends, I'd also run the Verifier on the sequence(s) in question to make sure there isn't a conflict.

Edited December 17, 2012 by George Simmons

[OurMikeyRocks]

I have run the Verifier and it gives me a "blue" light. I don't believe this is the problem.

I will try replacing the cables, both into and out of this controller. That's a bit of a chore - crawling around on hands and knees under a rats nest of wires. I'll see what happens.

I struggle with the cable thing because none of the other controllers downstream from the problem exhibit any issues. Doesn't necessarily make sense, but I will give it a try.

Thanks, Mikey

[OurMikeyRocks]

OK, I replaced all 5 cables in the megatree controller complex with brand new Cat-5e cables. There are 6 CTB16PC controllers to run the tree and the one that is acting up is in the middle of the daisy chain.

Unfortunately, I was just outside and this didn't fix the problem.

Of course, this isn't some obscure string of lights somewhere in the background. It is one of the spiral strings in the megatree. It couldn't be more conspicious!

Any other ideas? If nothing else, I am going to move the two channels to spare channels and re-program 19 songs.

Thanks for the help.

Mikey

[OurMikeyRocks]

I switched the misbehaving channel to a spare today and then reset the channel configuration for all of my sequences.

The problem is gone!

Does this mean the problem is not in the sequencing, but in the controller???

Mikey

[OurMikeyRocks]

Well, moving channel 1 to a spare in another controller fixed that channel. Over several evenings of the show I have noticed other channels doing the same thing - spontaneously turning on and staying on. After watching it for a while, channel 16 seems to be a popular one, but there are others.

Remember that I replaced all of the Cat 5e cable3s before and after the misoperating controller. I also reset the controller and reloaded the firmware. In addition and by accident, I discovered the Network Preferences tool and the settings it offers. First, I discovered that the box in the Options tab was checked for "Use compatibility mode for old MC-P channel controllers." I unchecked that option since I don't think I have any of those controllers. I ran the show for an evening and couldn't notice any change. I got several phantom lights and everything else ran the same.

Just for those who might want to know my network consists of 1 DIO32 + 1 DIO8RLY, 2 CMB16D cards, 1 LOR1602W, and 15 self-soldered, self-assembled CTB16PC cards. One of the CTB16PCs is mounted in the factory plastic case. The others are all mounted as pairs in 12" x 12" x 4" electrical boxes. 14 of these work great - 1 has problems.

Next, I went to the LOR tab and changed the speed of the Regular Network com port from 57.6K to 115.4K. The "help" for this suggested it could be done for configurations that are short and have a high channel count - my Cat 5e cable lenght is <400 feet and the channel count is just under 300. I ran the show for the last couple of evenings and can't tell a difference. I still have some phantom lights and everything else is normal.

My plans are to limp along to the end of this year's show and then "shotgun" all four of the ICs on the bad controller away - the Com chip, the PIC, and both latches.

Anybody got any other ideas or suggestions?

[plasmadrive]

Bourbon? Perhaps Rum!

[plasmadrive]

Actually.. can you tell if this happens the same time in the same place on the same songs? If not then most likely not your programming. If so most likely your programming..

If it is completely random look for solder issues, comm chip or an iffy RJ45 connector on the controller board.

I had a problem that was similar but not the same and I wound up finding a tiny ball of solder that appears to have broken loose after an entire season of use. It showed up after I set up for this year. I assume it broke lose with all the handling for storage and reassembly of the display this year. It was just hanging out around some of the IC pins.. This was a factory assembled "good to go" controller and not a kit but worked for an entire year.

[Chris B]

I have phantom channels when the computer is completely off. My wife noticed it the other day when leaving for work in the morning. Computer completely shut down, as I shut it down at the end of every show at night. This doesnt affect my show when its running that I've noticed but just kind of interesting where the phantom commands are coming from.

[OurMikeyRocks]

Actually.. can you tell if this happens the same time in the same place on the same songs? If not then most likely not your programming. If so most likely your programming..

If it is completely random look for solder issues, comm chip or an iffy RJ45 connector on the controller board.

I had a problem that was similar but not the same and I wound up finding a tiny ball of solder that appears to have broken loose after an entire season of use. It showed up after I set up for this year. I assume it broke lose with all the handling for storage and reassembly of the display this year. It was just hanging out around some of the IC pins.. This was a factory assembled "good to go" controller and not a kit but worked for an entire year.

I agree with your assessment, but, unfortunately, it is random. It does it in different songs, at different times, and on different channels. The other suffestions are certainly something I need to check.

The show comes down on the 2nd, so I will do a detailed inspection of the controller to see if I can find anything.

Thanks. Mikey

[Max-Paul]

Seems that a lot of us are having this problem this year. Found one cause and it seemed to clear it up for a bit. My one found problem was that I make all of my own Cat5 cables. I also have installed a box and in it a socket to terminate out near my first controller. Thus during the off season I dont have a cord just hanging from the eves. Well I made the mistake of wiring it wrong and a much needed ground wire was not there. This allows noise to interfere with the signal. So, things where looking good. Until I added a few new elements to the display, a Snoopy flying his dog house and 3 of those polar bears, all ican lights. I usually have all LED, will be changing these elements to LED this summer. And I had a queer thing happen with one of my controllers. I was showing a friend one controller and I just barely touched the comm wire. And the LED went out. But thought it blinked and so I freaked that I had a bad cable or socket. Took a few min for it to sink in to my thick head that it actually went out. I am going to pull the board and look for cold solder joint on the power side. But wondering if this is causing a loading shift on the RS-485 side that might be causing noise to be created and then misunderstood by one or more of the controllers turning on something that should not be turned on.

Weird and fun stuff to trouble shoot. And in case anyone missed it, that was sarcasm.

[OurMikeyRocks]

I didn't have any problems like this last year. I added four new controllers this year - one DC and three AC. They are kind of scattered through the RS-485 daisy chain physically, but they occupy the upper addresses in the network.

Is there any kind of termination necessary on the end of the daisy chain?

I believe that the controllers are not RS-485 repeaters, but just a pass through. That's why either jack can be the input and the other the output. Correct me if I am wrong in this assumption.

However, if this is true, then I don't understand why a bad/noisy cable would only affect the next controller. Why wouldn't it affect all controllers that are downstream from it (the bad cable) physically? Is there an affordable tester for Cat-5e?

I also looked at the data sheets for the ICs. The comm IC looks like a simple transceiver that just passes the signal along to the PIC. It looks to me like the PIC is the one to determine if the data on the wire is for it's address or not. So the PIC makes the decision to act on a command (because it is the right address) or not (because it is someone else's address). The latches seem to be exactly that, flip flops. With that being the case then I assume the PIC is also doing the twinkle/shimmer operations by setting and clearing the latches. If my logic holds water, then the PIC thinks it has received a command with it's address and command codes to turn a channel on with or without twinkle (I have seen both). Max-Paul, you seem to have some knowledge of the inner workings of the controllers. Are my assumptions reasonable?

I'm not sure how to interpret all of this, but these are my current assumptions based on what I have seen. Feel free to correct any of my assumptions and/or conclusions.

I want to see if I can reproduce this after I take the display down. That will make it much easier to find. I hate these witchcraft/Vodoo problems.

Thanks for the help. Mikey

[-klb-]

One comm chip might be more inclined to read noise as data than others. Or a nearby controller may be talking when it is not supposed to.

As far as the PIC, you may only be able to get it with the right boot loader from LOR. But I think the comm chip or a bad solder joint are both much more likely.

[Max-Paul]

Mikey, First I would like to say hello. Another thinking person, yahoo.

I agree with your assessment, sounds about right. I did a reverse engineering of the board for my own trouble shooting. Well the wiring part, have not looked up the chips other than who is wired to who and sounds like you and I have the same basic understanding what is doing what. I have bought a few boards each year for the past 4 years. This year I added 3 of the PC AC boards and one of the DC boards. And like you my new boards have the highest addresses, but in the middle of the daisy chain. The order of my string is 1, 2, 3, 6, 5, 21, 22, 23, 4, 7, 0A, & 0B. Rarely is there a frozen channel on 5 or 6. But is more common on 4, 21, 22, or 23.

At work some years ago, we had a problem with a temperature controller for industrial ovens. The 3rd oven was going crazy. Turns out it was the 2nd oven's controller was yapping when it should have been quit. On RS-485 the slaves (anything that is not the controlling computer is a slave) only talk when spoken to (addressed by the computer). I have pointed out in earlier post in different threads. Connectors are great for certain applications. One of them is for the comm chip. But also this can come with drawbacks. Oxide can grow between the pins of chips and the socket. Taking a small screwdriver and go and inch it up a bit slowly from side to side. Just before the chip comes out, push it back down. Might do this twice. This allows for the contacts in the socket and the pins on the chip to wipe off any oxide at the contact point. Use this as one of the last resorts. This can and in most cases start to weaken the springyness of the contact. If it was not a common chip to fail all together, I would be suggesting that it was soldered directly to the board. I cant tell you how many problems I have seen over the years due to sockets and multi pin plugs.

[OurMikeyRocks]

M-P

I found this on the LOR Accessories thread.

"RS485 when used in industry, is a "terminated" signal. The idea is that all the units present a high resistance to the data connection (as the LOR units do), then a 120 ohm resistor is placed across each end of the data pair. At low data speeds this is not always necessary, but once the speed increases significantly, it wil help reduce inteference and therefore errors in the data. I must admit though that the LOR communications are very robust, and most of the time you can get away without having to worry about termination resistors.

Regards,

Alan C."

Is there any chance this might help? It would be pretty easy to implement and then just plug into the last controller. Would there be any downside to doing this?

Mikey

[a31ford]

OMR, this last post (your's above mine) is where I'm heading on this, I've done much work with RS485, and ground loops (Yes I know ppl, they are 2 wire cords....) I have a post somewhere on the forum about a problem that sounds just like yours.

 

cable gauge, distance, even twist rate of the "twisted pair" (BTW each pair in cat 5 is at a different twist rate), as well as how close to the AC cords, ALL of these factors matter..... 

 

Try the 120 ohm, terminators, (BOTH ends), do NOT "Y" the cable at a controller, (rg; sending a 3rd cable to a lone unit) the cables must "as you say" daisy chain, one in, one out, director or PC, should be one end (with a 120 ohm resistor) the last controller in the chain should be the other 120 ohm resistor.

 

I made 2 short patches (M to F) with the resistor in the Female connector, this way ALL the cat 5 cables are NOT "special", just the 2 short adaptors, and the far one can be placed on the end of any cable, for the last unit.

 

Greg

Edited October 10, 2013 by a31ford

[Mark Birsching]

I had a problem like that last year, and it ended up being the PIC chip in one of the controllers was not quite seated (one pin must have had an intermittent connection). Caused a lot of havoc in the entire chain.  Was an intermittent problem, but one I found it, it's been fixed.  This was a G2 controller I had put together myself, so if you have G3, this would not apply. It would sometimes light up lights when the show was not running (or when I went to work in the morning). Long and short of it is it could be a controller issue - have you tried taking that one out of the chain to see if the problem stops?