Random pixels go out each time a show is run....

[LightsonOldPond]

Running 2 a/c controllers (G3), 2 Pixie 16s and 4 Pixie 8s via G3-MP3 Director.  I have 2 networks (Reg and AuxA) at 500k.  Reg network has 2,900 pixels and AuxA has about 1,800.   Each time I run a show a new portion of a string goes out, actually pixels on multiple strings go out.  It's not pixels on the same controller or even the same network each time.  If I stop the show and test a prop that has pixels out in HU they act the same (don't work).  I replace the pixels or splice in new ones and it all works in HU.  Fire up the show again and that prop will work but another totally random section of another prop will be out.   

About three days ago I found that a rabbit had chewed 90% through one of my Cat5 cables.  I replaced that cable but not sure if that would have fried something somewhere in one of the controllers?  Today I went out and did a reset of all my controllers (a/c and pixie) and that didn't seem to fix the issue.  Replacing pixels everyday is obviously not the answer so can anyone think of what else I can check/test?  It seems like it could be a power issue, but not sure what would cause the pixels to go bad rather than just not power on.  Almost ready to shut down for the season.

[k6ccc]

It's unlikely that Bugs did any real damage to anything except the Cat-5 cable.

2,900 pixels on a 500K network is really pushing it unless none of your sequencing is changing very fast.  If that network is only driving Pixie controllers, I would bump it up to 1000K speed.  However that is not causing your problem with pixels dying.  It sounds like you are having a high failure rate for your pixels.  A couple things to check.  First and foremost is what voltage are you feeding to the pixels?  Measure it with zero pixels lit, don't assume the answer.  Are they 5V or 12V pixels (yes, we have already had on person who was feeding 12 volts to some 5 volt pixels - it killed them.   Has it been raining?

Unfortunate reality is according to many reports, pixels from almost all vendors the past couple years has gone way down.  I have experienced that myself.

 

[LightsonOldPond]

Jim - appreciate the feedback.  They are 12v pixels, received an order of 10k last year but only used a couple thousand with no issues at all.  No changes this year other than adding more pixel props.  I've had more randomly go out this evening.  Some props have pixels that are sticking on various colors and when disconnected and reconnected the ones that were stuck are out for good.  I also had some props where as little as a few or up to 50% of the pixels go out, or at least when I went to check on them they were completely out. 

I have 2 G3 a/c controllers, 3 Pixie 8's and a Pixie 16 on Reg Network @ 500k.  Aux A has 1 Pixie 8 and 1 Pixie 16 @ 500k.  I can try to move another Pixie to Aux A but I'm losing pixels at a high rate from controllers on both networks.  Should I be checking power supplies, fuses, etc?  I'm not maxing out the # of pixels per port but it would have to be something related to the power and be the same issue with multiple controllers since it's happening on both networks, right?  As you mentioned, the network speed would cause lag, maybe pixels not coming on, but certainly not kill them.  I'm going to try to run just the network with the 2 Pixie controllers at 1000k and see what happens.  I'm at a loss, hoping to find the cause before shutting down for good.

 

 

[TheDucks]

Old Pond. 

Improper cleaning of the circuit board leads to corrosion over time. Look close. Is there grey powder around the solder joints.

[LightsonOldPond]

Thanks for the feedback.  I'll take a look as I go through them the next couple days.  For what it's worth, all of the Pixie controllers were brand new last year, stored inside in the off season. 

[k6ccc]

I think TheDucks is talking about the failed pixels - not the controllers.

 

[TheDucks]

10 hours ago, k6ccc said:

I think TheDucks is talking about the failed pixels - not the controllers.

 

True. I have seen this before on cheap stuff with limited QA steps.

To be honest, I can't conceive of a Controller failure that would even work close to as described.  Work or Fail.

It is a deck of command cards being passed to the nodes .Take the top one and pass the rest on (repeat 49 more times). When a node fails, the deck is dropped

[dibblejr]

I saw a 50% or greater pixel failure rate in both 21 and 22 from pixels purchased that were / are what I label as covid year pixels. That is why I recommend purchasing from LOR since they have a great warranty policy. Yes I lost a ton of $ since LOR was not providing the pixels I needed in the past. I had gift pixels sent to friends that were DOA. Either entire strings or partial strings.

Just remember there were chip problems all over the world.

JR

[LightsonOldPond]

Bad pixels def seems to be part of the issue.  Seems odd that they are all going bad at the same time, but they are from the Covid time period so there’s that.  Good thing is all 10k were free, bad thing is I may have gotten what I paid for lol.  
 

Today I checked all the Pixie power supplies, all were 11.98-12.  Tested all the fuses, all were good.  Double checked wiring inside Pixies, all good.  Noticed a couple of props start up with only a few pixels on, then they were good after a few sequences.  Another prop did the same thing then went out permanently except for a few pixels.  Odd stuff happening outside of just bad pixels.  I’m going to try to move one of the Pixie 8’s from the Reg network to the Aux network since I’m pushing the 3k limit on Reg.  Other thing I wanted your guys’ expertise on was the power supplies. I’ve mapped out how many 12v pixels are on each controller/port.  Pixie 8’s have 325 watt ps and the Pixie 16’s each have two 350 watt ps (1 to each bank of 8).  I don’t think I’m pushing the power supplies but can you see any reason to add a power supply anywhere?  All pixels are run at 30%.
 

[k6ccc]

Yea, your Regular network is really pushing it for 500K - and you can't go faster because of the CTB16 controllers.  Just doing a quick count, I would put the two Pixie 16s onto the Aux A network at 1000K and put all four Pixie 8 controllers along with the two CTB16 controllers on the regular network at 500K.  You may want to leave one of the Pixie 8s also on the Aux A network.  You have almost 4800 pixels (14,400 channels) between the two networks.  I would arrange it so that about twice as many channels are on the 1000K network as are on the 500K network.  I'll let you add up channels to see how that works best.  Might even want to run a third network with the CTB16s on their own slower network and then run both Pixie networks at 1000K.  In that case, roughly divide the controllers so the two Pixie networks each have about half of the Pixie channels.

 

[TheDucks]

Barely 12V at the Pixie input 🥴 is not good , and really poor if the show is not using a lot of lights. ON.at the time of measurement

(but probably not the cause of the random only color shifts near the end, unless they drop lower under load)

1) is the wire size from the PSU 12 or 14 Ga (minimum)

2) if the PSU is adjustable? Crank it up to 12.6 (+5% of a ±10% tolerance), a very safe level. as high as 13.2 is probably OK (no lights on).

You want better than 10.5 at the end of the string with normal lights on

BTW What is the output WATTS of the PSU feeding the problem controller? It may be collapsing under hard loads (lots of White)

And that Pixie 16 need 1 (300W+) PSU per bank. You might try a bit more Load Balancing on the banks (draw a Bank divide line on your chart, make the numbers closer to balanced bank 1 <> bank 2)  FWIW I don't know how to get S5/6 preview to auto do that Also, try putting the highest counts closer to the Power Feed end of the Pixie

💡 LOR, why not change the 16 port (and CMB24) layout and put the POWER FEEDS between ports 4+5, 12+13 for more power capacity with no additional copper(except around the feed lugs)?

[LightsonOldPond]

10 hours ago, k6ccc said:

Yea, your Regular network is really pushing it for 500K - and you can't go faster because of the CTB16 controllers.  Just doing a quick count, I would put the two Pixie 16s onto the Aux A network at 1000K and put all four Pixie 8 controllers along with the two CTB16 controllers on the regular network at 500K.  You may want to leave one of the Pixie 8s also on the Aux A network.  You have almost 4800 pixels (14,400 channels) between the two networks.  I would arrange it so that about twice as many channels are on the 1000K network as are on the 500K network.  I'll let you add up channels to see how that works best.  Might even want to run a third network with the CTB16s on their own slower network and then run both Pixie networks at 1000K.  In that case, roughly divide the controllers so the two Pixie networks each have about half of the Pixie channels.

 

Unfortunately I’m maxed out at 2 networks using a G3-MP3 - def something to upgrade for next year.   

[TheDucks]

3 minutes ago, LightsonOldPond said:

Unfortunately I’m maxed out at 2 networks using a G3-MP3 - def something to upgrade for next year.   

You might push the Pixie 8 with 4 unused ports to the other network (assumes that is NOT the one with issues)

Power budgets are fun 🙄 normally. You add up the loads. OTOH we rarely have an all white load (spec sheet). Nothing wrong with using that number other than $$$ for rarely use capacity. Got a Harbor freight handy? They have a DC clamp-on Ammeter  for under 100 (DC is trickier than AC for clamp on) or go 'old school' and get a automotive (0-60) Amp Gauge like https://www.amazon.com/Ammeter-Guage-Voltmeter-60-0-60A-Vehicles/dp/B08CVN6VQW  Use 10Ga so to not introduce additional drop. To see how close you are getting to your PSU limit while the show plays (the pointer will be moving about, so watch for peaks