Pulling my hair out!! Controllers dropping out constantly!

[dave58921]

I am having problems with CCP controllers losing connection with the PC. It is generally controllers somewhere in the middle of the line, start and end controllers all seem to work perfect and I have checked all of the CAT5 lines with a CAT5 tester and all cores are continued from the start of the line through til the end of the line. So I can’t see the CAT5 being an issue.

 

I only have CCB or CCP controllers on the lines that have issues:

 

Regular – 9 x CCP or CCB (3 or 4 controllers dropping out regularly)

Aux A – 8 x CCP (2 controllers dropping out now and again)

Aux B - CCR, CCP, CCR, CCP, CCR, CCR, CCP, CCR, CCR, CCR, CCR, CCP, CCP (line appears to be working fine)

Aux C – 14 x CCP (around 6 boxes dropping out constantly)

Aux D – 11 x CCP (around 4 or 5 boxes dropping out constantly)

Aux E – 19 x 50w flood (not had much time to test, communication seems okay but floods not responding correctly)

 

The problems seem to be caused by faulty controllers as, for example, I changed 2 x CCP controllers on the Aux B line and that solved the data problems I originally had. I also changed controllers on the Aux A line which improved data issues there.

 

The problem is there is no logic to the problems. If I change the controllers that are dropping out, the new controllers still drop out. For example I changed both controllers on Aux B that were dropping out and they continue to drop out, just those two so it has to be other controllers causing the issue.

 

The next issue is I have now changed upwards of 12 controllers and no longer have any spares, so it’s pointless continuing to swap controllers out.

 

I really need some advice on this as I am pulling my hair out trying to get this show going for my client.

 

Thanks in advance for your advice.

[k6ccc]

This may sound like a dumb question, but what do you mean by "dropping out"? Are lighting commands being missed, or the status light on the controller going to flashing, or something else.

What speed are the circuits, and are they enhanced or normal?

Sent from my Droid Turbo via Tapatalk, so blame any typos or spelling errors on Android

[dave58921]

Thanks for your reply.

The status lights are going from solid green to flashing on the controllers that are "dropping out".

High speed dongles used on all 6 networks, set to 500k and enhanced network. I have also tried at lower speed but no difference.

All controllers are on latest firmware and showing in hardware utility (except for when the status lights are flashing).

My gut tells me that it is one or two controllers in each line causing the issue, but at the same time based on what I've read and been told by LOR I don't believe the failure rate is anywhere near that high as half of the controllers are new out of the box and the other half were working fine for 9 weeks last season.

 

[dgrant]

With that many commands being sent, multiple paths, multiple RS485 connections...I'm wondering if your computer is able to keep up? You could have reached its limit and past it by a ways. My thoughts revolve around the computer architecture and just how fast it can talk to that many USB buss interfaces along with its normal processing

[dave58921]

4 hours ago, dgrant said:

With that many commands being sent, multiple paths, multiple RS485 connections...I'm wondering if your computer is able to keep up? You could have reached its limit and past it by a ways. My thoughts revolve around the computer architecture and just how fast it can talk to that many USB buss interfaces along with its normal processing

Thanks for your reply.

The issues are there even when just telling the controllers to be solid colour on one line. The problem is to do with data communication between boxes.

[khawes]

You might try switching out the cat-5/6 cables,  Though I would suspect a bad cable would stop communication down the rest of the line.  

Do the cat-5/6  cabled run in a bundle for the computer?  maybe there is cross talk?  Do they run next to a high voltage power line?

Are the USB to RS-485 connectors using a USB hub, if so try plugging one of the bad networks into the computer directly, if that works replace the bub or perhaps the hub is overwhelmed and you need to move to more than one.

... and with that I'm out of ideas.

[wbaker4]

You must be using USB Hubs to connect all of those dongles to your PC.  What level USB do the HUBs support?  2.0 (Hi-Speed)?  3.0 (Super-Speed)? 

If it supports 3.0, I would suggest that you go out and find a 2.0 HUB and try that.  Not sure what the RS485 dongles support, but using a lower speed hub may solve your problem.  Perhaps even USB 1.x HUB.

If you have all of the USB dongles plugged in to your computer, I would suggest buying a few external hubs.  Make sure they are self powered too.

Don't plug the dongles into any of the blue (USB 3.0) USB ports.

Walt

[dave58921]

2 hours ago, khawes said:

You might try switching out the cat-5/6 cables,  Though I would suspect a bad cable would stop communication down the rest of the line.  

Do the cat-5/6  cabled run in a bundle for the computer?  maybe there is cross talk?  Do they run next to a high voltage power line?

Are the USB to RS-485 connectors using a USB hub, if so try plugging one of the bad networks into the computer directly, if that works replace the bub or perhaps the hub is overwhelmed and you need to move to more than one.

... and with that I'm out of ideas.

The issues on each line still occur when just that one line is plugged in. The PC has 5 USB outputs (3 x 2.0 and 2 x 3.0) I'm using 4 of the PC's USB for the busier lines and then the floods and one of the other lines is in a hub.

The CAT5 is running in close proximity to 110v power that's supplying the controllers, but the CAT5 cable is Van Damme SF/UTP Tactical CAT 5E foil and braided with Sentinel shielded connectors so I don't think that is the issue. We ran the show for 9 weeks last year with no issues (after we had solved fewer but similar problems to this year) and no data lag.

 

[dave58921]

1 hour ago, wbaker4 said:

You must be using USB Hubs to connect all of those dongles to your PC.  What level USB do the HUBs support?  2.0 (Hi-Speed)?  3.0 (Super-Speed)? 

If it supports 3.0, I would suggest that you go out and find a 2.0 HUB and try that.  Not sure what the RS485 dongles support, but using a lower speed hub may solve your problem.  Perhaps even USB 1.x HUB.

If you have all of the USB dongles plugged in to your computer, I would suggest buying a few external hubs.  Make sure they are self powered too.

Don't plug the dongles into any of the blue (USB 3.0) USB ports.

Walt

Hi Walt,

The PC has 2 x 3.0 USB and 3 x 2.0. I'm using a hub to run the floods and 1 other line. The rest are direct into the PC.

The problems with controllers dropping out occur even with just 1 network is plugged in, which should rule out the PC not being able to handle the data rate.

interesting idea with the USB 3.0 potentially being an issue, but when testing one network plugged in at a time some of which would be on the 2.0 USB, I still have the same controllers dropping out.

Is there any chance that CAT5 could be an issue? Even though I have put a tester on the PC side with the remote tester at the last box on each line and had continuity on all cores? (Well cores 4,5,6 and 8 which LOR uses)

Edited October 25, 2016 by dave58921

[wbaker4]

Do you have PRO level and have the enhanced network checked for 1000K operation?

I assume your dongles are all High Speed?

You have 55 CCR/CCB devices.... not sure what the limit is for those devices on the LOR network.

[DownTown]

49 minutes ago, dave58921 said:

The CAT5 is running in close proximity to 110v power that's supplying the controllers, but the CAT5 cable is Van Damme SF/UTP Tactical CAT 5E foil and braided with Sentinel shielded connectors so I don't think that is the issue.

That foil and shield on your UTP will simply act as an antenna to pick up interference unless the ends of the foil/braid are properly grounded. If  you are simply connecting the 8 wires inside end to end and leaving the shield un-terminated/un-grounded, you are inviting problems. 

D.T.

Edited October 25, 2016 by DownTown

[k6ccc]

1 hour ago, wbaker4 said:

Do you have PRO level and have the enhanced network checked for 1000K operation?

Can't use 1000K speed.  CCP/CCB/CCR does not support 1000K - only the Pixcon does.

 

[wbaker4]

3 hours ago, k6ccc said:

Can't use 1000K speed.  CCP/CCB/CCR does not support 1000K - only the Pixcon does.

 

Yes - I forgot about that!

[dave58921]

13 hours ago, DownTown said:

That foil and shield on your UTP will simply act as an antenna to pick up interference unless the ends of the foil/braid are properly grounded. If  you are simply connecting the 8 wires inside end to end and leaving the shield un-terminated/un-grounded, you are inviting problems. 

D.T.

This is a possible issue. Does crimping the metal shield of the cat 5 plug to the cable braid do the job or does something else need to be done?

[DownTown]

3 hours ago, dave58921 said:

This is a possible issue. Does crimping the metal shield of the cat 5 plug to the cable braid do the job or does something else need to be done?

I doubt that Light-O-Rama devices are designed to take advantage of the metal shielding of your connectors. If memory serves me (I'm not in front of my controllers), the RJ-45 jacks are all plastic, and so there is no ground for the shield to connect to when you plug your cable in. The shielded cables/connectors are designed to be part of a complete system that is engineered to take advantage of them.

D.T.

[dave58921]

6 hours ago, DownTown said:

I doubt that Light-O-Rama devices are designed to take advantage of the metal shielding of your connectors. If memory serves me (I'm not in front of my controllers), the RJ-45 jacks are all plastic, and so there is no ground for the shield to connect to when you plug your cable in. The shielded cables/connectors are designed to be part of a complete system that is engineered to take advantage of them.

D.T.

I had the same thought today. They are plastic and thinking about it, wouldn't the unit itself need grounding for the shielded CAT 5 to have any affect? The CCB, CCR and CCP controllers all have no ground wired to them.

As using S/FTP cable has potentially created problems rather than solving them, in theory could I just solder a cable between one of the CAT 5 connectors on each CAT 5 cable to the ground on the power line? I have used 3-core flex to run the power to each controllers, so there is a ground available, it's just the ground is not connected to any of the Cosmic Color devices, but is used to ground the floods.

The good news is that 90% of problems have been solved by putting a 120 ohm resistor across the data lines at the end of line, as suggested by LOR last night. I should have thought of doing this, but I don't have much experience with CAT 5 other than last year's show which we had fewer but similar problems and we solved them without using a terminator.

Thanks for all of your help and hopefully we will get the show 100% problem free by Tuesday, when the Halloween shows end and the more important (to the client anyway) Xmas shows begin.

 

 

[k6ccc]

Agreed about the shielded cable.  It does no good if it's not grounded.

As for the termination, I don't know why I did not think of that.  RS-485 standard specs that the line should be terminated at BOTH ends.  On a short and simple network, it's not much of an issue, and MANY people get away with terminations just fine.  However, as the complexity of the network increases (and yours is somewhat complex), and the speed of the network increases, it can be more of an issue.  I built up a bunch of RJ-45 terminators for the "far" end of the line that terminate both the pair that is used for LOR networks and the pair used for DMX networks.  I'm not using DMX, but it was easier to just put both resistors in the connector and not have to worry about which type of network the terminator was going onto.  For the USB-485 end of the cable, I built up a few connectors that have am RJ-45 male on one end and RJ-45 female on the other end with less than one inch of wire in between.  Again, both pairs are terminated.

 

[dave58921]

4 minutes ago, k6ccc said:

Agreed about the shielded cable.  It does no good if it's not grounded.

As for the termination, I don't know why I did not think of that.  RS-485 standard specs that the line should be terminated at BOTH ends.  On a short and simple network, it's not much of an issue, and MANY people get away with terminations just fine.  However, as the complexity of the network increases (and yours is somewhat complex), and the speed of the network increases, it can be more of an issue.  I built up a bunch of RJ-45 terminators for the "far" end of the line that terminate both the pair that is used for LOR networks and the pair used for DMX networks.  I'm not using DMX, but it was easier to just put both resistors in the connector and not have to worry about which type of network the terminator was going onto.  For the USB-485 end of the cable, I built up a few connectors that have am RJ-45 male on one end and RJ-45 female on the other end with less than one inch of wire in between.  Again, both pairs are terminated.

 

Interesting idea with putting a resistor at the USB end, I've not heard of that being done before. Would you advise doing that for all 6 networks on this show? How would I go about getting the resistor in the chain at this end? For the end of line I just wired a CAT5 one end of a short cable and then twisted one end of the resistor with core 4 and one end of the resistor with core 5 of the cable, but with the start of line it's not that simple.

[k6ccc]

1 hour ago, dave58921 said:

Interesting idea with putting a resistor at the USB end, I've not heard of that being done before. Would you advise doing that for all 6 networks on this show? How would I go about getting the resistor in the chain at this end? For the end of line I just wired a CAT5 one end of a short cable and then twisted one end of the resistor with core 4 and one end of the resistor with core 5 of the cable, but with the start of line it's not that simple.

I'm going to preface this with something.  As I said, the RS-485 standard states that the data line should be terminated at both ends of the cable.  The LOR controllers are not terminated - nor should they be since in many (or most) cases there is another controller further down the circuit.  I have never heard LOR state if the RS-485 adapters have a termination built in.  In this case, it would make sense since the adapter is normally at the end of the circuit.  However I have assumed that there is not a terminator built into the RS-485 adapters, and therefore added one.  If LOR comes out and states that the RS-485 adapters have a built in terminator, then obviously there is no need to add one.

For the end at the RS-485 adapter end of the circuit, what I did was to take a female jack connector commonly called a keystone jack, and did not mount it into the holder that it is normally mounted into.  I wired a short Cat-5 cable to the wiring part of the keystone jack, and then put a RJ-45 male plug onto the other end of the short cable.  The length of cable was about one inch.  I then added a 120 ohm resistor across the appropriate pins of the keystone.  Because the keystone jack is not designed to have two wires (in this case one wire and one resistor lead) on a single connection point, I carefully soldered the resistor connection.  Keystone jacks are available from Home Depot (along with MANY other places):

http://www.homedepot.com/p/CE-TECH-Category-5e-Jack-White-5025-WH/204410615  or in a package of 10

http://www.homedepot.com/p/CE-TECH-Category-5e-Jack-White-10-Pack-5025-WH-10/204410618

The result of this is a very short cable with a female jack on one end and a male plug on the other.  The plug goes into the RS-485 adapter and the cable to the first controller goes into the jack.

You could also do a similar project using a surface mount jack (sometimes called a "biscuit").

http://www.cablewholesale.com/products/network-phone/surface-mount-jacks/product-300-313se.php?utm_source=GoogleShopping&utm_medium=cpc&utm_term=300-313SE&utm_campaign=Cat5e Single Surface Mount Box%2C Unshielded&gclid=Cj0KEQjwqMHABRDVl6_hqKGDyNIBEiQAN-O9hDYzidG38RIWdrbLGv9Mb3OwXSqq7fvKrpX4Cr5-W3oaAsmB8P8HAQ

Some of those have screw terminals rather that insulation displacement punchdown connections - which are easier to wire (although the link above is not screw terminals).

 

[wbaker4]