P5 panels freezing

[rick gurnee]

I'm having a small performance problem with my P5 panels and I was hoping I could get some ideas on where to look. Here is my set up. I have two p5 displays. The first is 3x2 (prop in preview is a matrix with 64 strings and 192 nodes per string). It is controlled by a raspberry pi 4 with FPP and a color light card with a static IP address of 192.168.0.101 and uses universes 101-172. The second is 6x7 (224 strings with 384 nodes). It has a pi 4 with FPP and two color light cards with a static IP address of 192.168.0.102 and uses universes 172-676. They have separate cat6 cables plugging into a 1G Ethernet switch.

I have a 30 second test sequence that I am running that displays several pictures on the large display and scrolling text on the small display. When I run this the screens will freeze for a fraction of a second every 10-15 seconds.

I ran in both the Sequencer and as a show and the same freezes were seen. The sequencer used considerably more CPU during the run than when I ran it as a show. Note, all of the discussion that follows was when I ran it as a show.

I then exported this to a FSEQ file and uploaded it to FPP and it ran perfectly. I then used Xlights to run the FSEQ file and it also ran perfectly.

I don't have a lot of tools to monitor what is happening, but I have included 4 screenshots that gives me a little insight. The first pair are the FPP status screens for the LOR run and the Xlights run. At first I was concerned about the errors I was seeing. Universes 173-310 never show any errors while universes 311-676 show an error every 10 seconds or so. However, the Xlights screen also shows errors on universes 311-676, although fewer, and it ran fine. I would like to further investigate these errors, but I don't think they are causing the freezes. I tried looking in the FPP logs for these, but I couldn't find them. I will further investigate.

The second pair of screenshots are from the Task Manager showing the Ethernet activity. Here you see the Xlights run stays steady at 63 Mbps while the LOR run has many dips which correspond to when I saw the displays freeze. Also, there was wi-fi data of approximately 8 Mbps which had dips at the same times as the Ethernet dips. (I believe the wi-fi data shouldn't be happening and I believe LOR is already aware of this). I don't believe the wi-fi data is causing the freezes.

The CPU was running at less than 10% during this run and there was no CPU spike during the freezes.

So I'm hoping someone can give me suggestions on how I can further investigate why these freezes are happening. Clearly they are related to the dips in Ethernet traffic, but is that the cause or another symptom.

In an unrelated point, I believe LOR is aware that after a sequence finishes, data is still being sent (in my case about 63 Mbps). It turns out that Xlights also continues to send 63 Mbps out after the sequence is finished. In Xlights, turning off the button to control lights stops this. In LOR, shutting down the control panel stops this.

Here are the links to the screenshots:

https://www.dropbox.com/scl/fi/spit4681jxxs6njirufmy/LOR-FPP.jpg?rlkey=lm641zvcs6y4mtzwj4yzanfpe&st=y6t2dcmm&dl=0

https://www.dropbox.com/scl/fi/a1jikt2fpk7fj3mgzgoeo/LOR-TaskManager.jpg?rlkey=54y67w30i99vi5adk7356cko4&st=n6639xrn&dl=0

https://www.dropbox.com/scl/fi/3wr2sl9fa9v6k8xtsyhdu/xlights-FPP.jpg?rlkey=elowdk3bf6xolzgemsv191tc4&st=ngu2ndfz&dl=0

https://www.dropbox.com/scl/fi/4b7accoraavs1doch9z12/xlights-TaskManager.jpg?rlkey=dl44ka15clwkq06p071fyycei&st=jfuztc73&dl=0

 

[k6ccc]

1 hour ago, rick gurnee said:

I ran in both the Sequencer and as a show and the same freezes were seen. The sequencer used considerably more CPU during the run than when I ran it as a show.

Interesting on the freezes.  And yes, the Sequencer should be taking a lot more CPU as compared to the Show Player.  That is to be expected.

 

1 hour ago, rick gurnee said:

The second pair of screenshots are from the Task Manager showing the Ethernet activity. Here you see the Xlights run stays steady at 63 Mbps while the LOR run has many dips which correspond to when I saw the displays freeze.

The LOR Show Player will switch universes that have no lights at that moment to just sending idle packets, so dips is fairly normal.  You can see that in the FPP packet counter display.  More likely to see that on the text display than the butterfly.  I don't know exactly how long it take of idle before it switch to idle packet mode, but it appears to be fairly short.  I see that regularly on my TuneTo sign where some rows will be black at times.

1 hour ago, rick gurnee said:

I believe the wi-fi data shouldn't be happening and I believe LOR is already aware of this).

It has been discussed here before and I also believe LOR is aware of it.  In my case it is two wired interfaces - not WiFi.

Most of the nitty gritty of this will likely be a @MattBrown issue...

 

[rick gurnee]

Well, dips may be normal, but in my case I saw there was a direct correlation between the dips and the freeze.  Never saw a freeze without a dip.  I can't be positive if I saw a dip without a freeze or not.  Also, for what it's worth, if you look at the Xlights Ethernet activity, you see a very steady line.

[rick gurnee]

A little more information.  I changed the LOR network setting for my large display to only use universes 173-310.  I did this for two reasons.  First, universes 311-676 were getting errors and I didn't think these were the problem, but I wanted to remove any doubt.  Second, I thought this freeze problem was related to the size of my display.  The large display has 86,016 pixels and the small one has 12,288 for a total of 98,304.  I'm reducing that to 35,840 (if my math is right).  This puts it more in line with the size of other people.

When I ran my test, of course only the top part of the large display showed anything (universes 172-310) and the small display showed everything (universes 101-172).  However, the freezes are still there!  And because I am not going beyond universe 310, no errors are showing up in FPP.  The task manager is showing about 22 Mbps and I still see dips whenever the freeze occurs.

I then reduced it even further to end at universe 175. So I only had one line of pixels on the big display, but the small display still showed everything.  The freezes are still there but they seem to be less frequent and don't last as long.  But they are still definately there.

[rick gurnee]

Well, I fixed the freezing issue.  And you're not going to believe what I did to fix it.  I decided to play with some of the color light parameters to see if they had any affect.  In order to adjust the color light parameters, I needed to unplug the cat6 cable from the pi and plug it into the first color light card.  I then ran LEDVision and decided to reduce the refresh rate from 1920 to 960.  I moved the cat6 cable back and tried my test.  NO FREEZING! Also, the data rate on the Ehernet went from 62Mbps (with dips) to a very steady 115Mbps.  I then tried some other refresh rates and they all seemed to work.  I then went back to 1920 and it work, too!  That really confused me.  I tried different DCLK rate, too and those worked until I got above 20.8 and I saw a lot of pixilating.  So I ended up setting the refresh rate to 920 and the DCLK to 20.8.

I then shut everything down and restarted everything and the freezing was back and the Ethernet data rate was back to 62Mbps with dips.  At this point I thought I was going crazy.  I moved the Cat6 cable and brought up LEDVision to verify that my parameters had stayed to the same.  They had.  So I ran a test again and everything worked!  Now I was completely baffled.  I hadn't made a single change and it started working.  And then I got my eureka moment.  I stopped LEDVision and the freezes started happening.  While my test was running, I started LEDVision and everything worked.  Stop LEDVision and freezes start!  I didn't have to do anything in LEDVison, I just started and the first menu came up and the freezes stop and the Ethernet rate goes from 63Mbps to 115Mbps.

I can't explain it, but that's what happens.  Maybe Matt has an idea.

[k6ccc]

Interesting!

 

[rick gurnee]

On 5/3/2024 at 11:22 AM, k6ccc said:

You can see that in the FPP packet counter display

It's possible that you could see dips in the FPP packet count display, but not in the Task Manager Ethernet Network display.

Here is what I have found.  Remeber, I have two displays, one with 6 panels and one with 42 panels.  When I start LEDVision, I see 53Mbps on the Ethernet network. LEDVision only knows about the 42 panel display.  So this works out to 1.26Mbps per panel.  This is happenong even on the very first menu screen of LEDVision before I even tell it where the panels are.  Therefore, I'm assuing it saved that information from the previous time I started LEDVison.

When I start Xlights (who is sending data to both displays) I get a very steady 62Mbps or 1.29Mpbs per panel.

When I start LOR (which is sending data to both displays) I get a very bumpy 62Mbps with a lot of dips which correspond to the displays freezing.

When I start LEDVision (and see 53Mpbs) and then start LOR, I see a very steady 115Mbps and no freezes (53Mbps from LEDVision + 62Mbps from LOR),

My question still is why am I getting freezes when I run LOR by itself and why do they go away when LEDVision is also running.

Is it possible, Jim, that you are getting the same thing but didn't notice it?  When I reduced the outout from LOR to just the 3x2 display, I still see the freezes, but they are reduced.  In fact, if I wasn't looking for them, I might not have noticed.  I just had text scrolling across the top and bottom of the screen.  However, if you look at the Task Manager and expect to see a very smooth consistant line, you will see dips every once in a while.  These don't occur when LEDVision is running with LOR or when Xlights is running.

I'll do more investigating, but I really don't want to have to run LEDVision and double my network load to avoid the freezes.

[k6ccc]

I'm confused by what you are saying.  Why would you ever have LOR or xLights running at the same time as LEDVision?  And what is LEDVision sending data to?  LEDVision only talks to the panels, and LOR and xLights only talk to FPP - on a completely separate network.  I mean you have to move network cables around in order to use LEDVision vs LOR or xLights.

 

[k6ccc]

And BTW, my knee is recovered enough that I can contort myself to get to my off season storage for my panels, so after a three month pause, I should be able to start TRYING to get LEDVision working and do some testing with it.

 

[rick gurnee]

I agree with you and it doesn't make sense.  Let me give you the details of what I did and what I found out.  Keep in mind that I agree you should never run LOR and LEDVision together.

I was having problems with my displays freezing when I ran LOR but not when I ran the same sequence (converted to FSEQ) in Xlights.  As I investigated, I noticed the dips in the Ethernet traffic that cooresponded to the freezes.  I thought maybe it had to do with some of the Color Light card parameters.  (I really didn't thing this was the problem, but I was grasping at straws.)  So I moved the Cat6 cable from the Pi to the Color Light card and ran LEDVision.   I made a small parameter change, moved the cable back to the Pi and tested. (note, by accident I left LEDVision running) Everything worked fine, so I thought my change in the paramter was what fixed the problem.  I then switch the cat6 cable back to the color light card and tried other parameter changed.  Each time I would move the cable back and run the LOR test.  Without thinking, I just left LEDVision running because I knew I was going to go back to it, and I didn't think it mattered.

Thinking I had resolved the issue, I set the color light card parameters to what I thought was best and dit one final test and everything worked fine.  I shut down LEDVision and ran one final test and the freezing came back!  Upon further invetigating, I realized that if I had LEDVision running, even without the cat6 cable plugged into the color light card, the freezinf went away.  I never even go past the first menu screen.  As I said, LEDVision must remember the previous settings because as soon as I start LEDVision I see 53Mbps of data going across my network.

For what it's worth, I just started LEDVision and the panels are not even plugged in, yet it's sending 53Mbps across the network.  I turned on the panels and tried to get LEDVision to detect the cards but it says "no cards detected" as it should.  But it's still sending 53Mbps across the network!

Glad to hear your knee is recovering.

[rick gurnee]

For what it's worth, I used verion 8.8 of LEDVision.  The latest, 9.2, confused me.  However now that I have used 8.8 a bit, I think I could get 9.2 to work.  They just moved some of the screens around.  But 8.8 works fine for the cards that I have.