Why is the LORCommPortListener so Chatty?

[jstorms]

So I've seen a few threads on networking issues due to load. I think to myself why is the LORCommPortListener so Chatty?

 

So for fun I thought I'd peel back the first layer of this onion. I start by going and downloading a free protocol analyzer (https://www.wireshark.org/)

 

I start by turning on the LORCommPortListerner by starting the LOR Control Panel on my computer on 192.168.1.200.

Not too bad. SSDP is a service discovery protocol and ARP is address resolution protocol. Basically LOR has these guys in its network preferences and it is trying to figure out what MAC addresses go with the IP Addresses. MAC address is the address a devices uses on its own little segment. Think of MAC as your face and IP as your name. Here my device .200 knows your name is .207, 208, or .210. It says hey anyone know .210? Then you say I'm .210 and it puts the face (MAC address) with the name (the IP address)

 

Like I said pretty tame stuff, and ARP does not/cannot route, so no problem here.

 

Next I turn on my E1.31 pixel controller on 192.168.1.210. This is where it gets fun.

 

This is the LORCommPortListener talking to the configured E1.31 device. Note I'm not running a sequence or anything. We are idle.

 

In 1 minute the Comm Port Listener running on my PC (192.168.1.200) sent 17422 of these packets. Each packet is 680 bytes. Totalling 11846960 bytes or 11.3 megabytes in 1 minute. In an hour this is 678 megabytes.

 

So I turn on a second configured E1.31 device that is in my network preferences. The traffic does not double, but does go up substantially. With two devices, in 1 minute I get 27379 packets * 680 bytes = 18617720 bytes = 17.8 megabytes/minute or 10680 megabytes/hour.

 

So what is going on?

 

ANSI E1.31 - 2009

Entertainment Technology – Lightweight streaming protocol for transport of DMX512 using ACN

 

This is built on top of E1.17

Architecture for Control Networks - ACN Architecture 

 

So the above diagram gives an overview of what is going on.

 

So I grab one of these 680 byte packets and this is what we see:

 

Now I haven't taken the time to breakdown the ACN protocol data units here (I do have a bit of a life). But at quick glance we see that 638 bytes of the 680 byte packet is data.

 

In these 638 bytes of data we can pick out ASC 1.17, and "Light-O-Rama Comm Listener" each of those letters take one byte. Then there are a few other bytes that are set which contain protocol information. But look at all the zero's. The bulk of what the LORCommPortListener is pumping out, thousands of times per minute, is mostly zeros. 

 

Next you take two packets and compare them side by side and there is virtually no difference at all, only a couple of bytes. I would venture a guess that these slight differences are probably increment sequence numbers in the protocol itself.

 

Next, all of this traffic is from the LORCommPortListener to the E1.31 device, never the other way around.

 

So my question is this, What is being accomplished by sending virtually the same, zero-padded packet over and over to the tune of thousands of packets/minute?

Edited October 5, 2015 by jstorms

[PhilMassey]

Seems to be quite a bit less chatty with 4.2.4 beta.

[k6ccc]

Very simple. DMX is very dumb. All it knows is a level set for each channel. It needs to be resent very regularly. So what is happening is every single universe is sent every 22 milliseconds. All those zeros is the level setting for all 512 channels in that universe (whether they are used or not). So each universe is the 680 bytes that you saw about 45 times per second. This totals about a quarter megabit per second per universe.

If I did the math right, your 17,422 packets in a minute works out to about 6 universes.

And yes, DMX over RS-485 behaves the same way (except for no IP routing bits), and it's only one universe per RS-485 interface.

[jstorms]

Yes, 6 universes! Thanks Jim! I was hoping someone more familiar would jump in. So the comm port listener is just following the rules of the protocol.

[k6ccc]

Correct.

 

However as Phil said, it's a lot less in 4.2.4.

 

This from the release notes for 4.2.4:

In some situations, when the lights for a DMX universe using E1.31 are all off, the Comm Listener will now deliver fewer E1.31 packets than it had previously.  This will result in less comm traffic on the ethernet network, especially during long stretches of time when the lights are all off (for example, idle time between shows).

 

I just upgraded my show computer to 4.2.4 and started my evening show.  I MUST point out that my evening show starts with a trigger which will happen in about an hour when it gets dark.  Therefore, the show is not doing anything at this time.  I then started WireShark and captured about 30 seconds of data from my E1.31 LAN.  WOW!  What a difference.  While the show computer is idle, the Comm Listener is spitting out the the 680 byte packets for the three universes that are currently in my network configuration, a little over once per second.  For example, to start my sample, universe 1 is sent at the following times (in seconds from the start of the capture):

0.000

0.820

1.642

2.464

3.285

4.105

4.925

5.746

6.449

7.386

 

Universes 2 & 5 showed similar time intervals.

 

I fully expect when the show starts, the traffic will go back to the "normal" 22mSec between packets.  It will be interesting to look because although Universes 1 & 2 are in the Network Preferences, they are not used in the year round landscape lighting show.  So it will be interesting to see if the only universe 5 goes to normal mode...

 

And before someone asks why Universes 1 & 2 are in the Network Preferences at this time even though they are not used in the Landscape lighting show, the light strips are there and I'm starting to do some testing of the Christmas show which DO use those two universes.  I just turned them on a few nights ago.

[k6ccc]

Yes, 6 universes! Thanks Jim!

 

Damn, I'm glad you were able to confirm the 6 universes part!

 

And I'll bet the second card had 4 universes...  (hence the reason the data did not double).

[k6ccc]

I fully expect when the show starts, the traffic will go back to the "normal" 22mSec between packets.  It will be interesting to look because although Universes 1 & 2 are in the Network Preferences, they are not used in the year round landscape lighting show.  So it will be interesting to see if the only universe 5 goes to normal mode...

 

 

Damn, I like it when a plan works.  My show started about an hour ago and I did another WireShark capture.  As expected universes 1 & 2 remained sending packets every .82 seconds, and universe 5 (the only one being used) went back to a packet every 22 mSec.

[LOR Staff]

So, we actually DO know what we are doing, huh?  Go figure

 

For the rest of you who want to continue to blame us, please check out what Jim (k6ccc) is posting.  He is mirroring everything we say or we mirror what he says.  And he doesn't work for LOR and has no reason to tell anything but the truth.  Not that we lie, but no one seems to want to believe us when we tell them that how things are is how things are supposed to be.

 

There are 2 main complaints we are seeing.  One is questioning the AMOUNT of data we are sending.  This thread is SUPER INFORMATIVE for that.  What has been described is EXACTLY what happens.  There are exact refresh rates for DMX, and we send packets at every refresh interval, because that is what is called for in the specification and we follow the spec.

 

There is a loophole in the E1.31 spec that allows us to turn down the amount of data being pumped out when the data is all 0s, which is what 4.2.4 is doing.  But frankly we only did that as a way to quiet down the complainers who don't understand how it works.  IMHO, the more correct way was how 4.2.2 and earlier did it.

 

Jim was absolutely correct when he speculated the amount of data would increase again once a show started.  When a show is running, you are going to go right back up to 22ish MS per frame.  Why?   THAT is the spec - https://en.wikipedia.org/wiki/DMX512 :  " A maximum-sized packet, which has 512 channels (slots following the start code), takes approximately 23 ms to send, corresponding to a maximum refresh rate of about 44 Hz."  We send a little bit faster than 23 to ensure the data reaches the bridge at the correct time.

 

As Jim has said, DMX is DUMB, but it is also self-correcting.  As per the spec, we have to tell the channel what level it is EVERY time the spec tells us to.  E1.31 uses UDP which is connection-less.  That means that a packet can be LOST or CORRUPTED at any time.  A lost packet in the DMX world will amount to a 'hang' of a channel for an average of 34ms (23 for the missing packet + about 11 on average to get to the re-transmitted byte).  The idea with sending all the 0's at idle was to make sure that even missed all-off packets would be corrected as soon as possible.  If the ending packet of a sequence is lost, it may now be up to 1.2 seconds before a missed all off packet is corrected (I think - I'm actually not sure how Bob coded the change for 4.2.4).

 

The other complaint is that packets are leaking out to the ISP, and or performance problems with networking.  99% of the time this is due to a bad user configuration, and 100% of the time it is outside of LORs control.  In an effort to try to explain how all this data gets around and how to prevent these issues yesterday we published "An Introduction to DMX and E1.31 for Pixel Control".  It is available on our website and here as a direct link: http://www1.lightorama.com/PDF/IntroductionToDMXandE131.pdf.  EVERYONE who does not understand networking should read that NOW.  People like Jim on the other hand should avoid it like the plague since it glosses over many very important topics in an effort to get you something that JUST WORKS.  I'll apologize in advance Jim, you WILL cringe if you read it!

[k6ccc]

Mike, I understand that last two sentences very well. I teach end users (cops, firefighters, other local government personnel) how to use their two way radios as part of my job. I regularly have to dumb it down to the level of the users so they will understand it. That often means glossing over details, etc.

And thanks for all the high praises. I guess maybe I do understand this stuff pretty well. At least I can read a WireShark capture.

[ed26105]

I put a second network card in the computer to run shows and keep it off the home network.  It wasn't that hard to set up and seems to be a good way to go.

[k6ccc]

I put a second network card in the computer to run shows and keep it off the home network.  It wasn't that hard to set up and seems to be a good way to go.

 

Correct.  It is pretty easy.  There is a little gottcha you should know about.  Make sure you have something connected to that second NIC.  If the second NIC is unconnected, Windows will route all your E1.31 traffic onto the other NIC in an effort to get it delivered.  Does not need to be an E1.31 controller, but it does need to be something - a switch will do.

[ed26105]

The second NIC is connected to a switch, then to all E1.31 devices.  LOR devices go through USB LOR networks.  No problems that I can tell so far, except needing to dial back the 12V power supply a little to help hanging pixels in E.1.31. 

[LarryDrumAZ]

The newest release has a "less chatty" comm listener.  Unfortunately, my license doesn't cover that version.  So I went with a 2nd NIC card to isolate the E1.31 show network from everything else.

 

Edit : I corrected my wording.

Edited October 15, 2015 by LarryDrumAZ

[bob]

What do you mean "LOR fibbed"?

[LarryDrumAZ]

When S4 was released from Beta. my license said it covered version 4.*.  When I checked with this latest release it said I am only good to 4.1*

[bob]

Where did you see something saying that your license covered "4.*"? We don't phrase things in those terms (at least not intentionally), because our licenses don't work that way.  We don't give out licenses that cover every possible X.*.* version, and we never have.  We always say "4.0.*" or "4.1.*" or "4.37.*" or whatever, not "4.*".

 

Can you find where it says "4.*"? For example, perhaps in an email that was sent to you? If so, please show it to me so I can fix it.

[LarryDrumAZ]

In 1 minute the Comm Port Listener running on my PC (192.168.1.200) sent 17422 of these packets. Each packet is 680 bytes. Totalling 11846960 bytes or 11.3 megabytes in 1 minute. In an hour this is 678 megabytes.

I ran Network Traffic View and got these results.  Now, granted I only have 1 E1.31 controller right now.

2,593 packets in 1 minute for a total of 1,654,334 bytes sent.  Same as you, idle with no show running, just the control panel.

Edited October 15, 2015 by LarryDrumAZ

[LarryDrumAZ]

Where did you see something saying that your license covered "4.*"? We don't phrase things in those terms (at least not intentionally), because our licenses don't work that way.  We don't give out licenses that cover every possible X.*.* version, and we never have.  We always say "4.0.*" or "4.1.*" or "4.37.*" or whatever, not "4.*".

 

Can you find where it says "4.*"? For example, perhaps in an email that was sent to you? If so, please show it to me so I can fix it.

 

I thought I was covered for S4 for the foreseeable future, but apparently I need to upgrade.  Somewhere I missed something and read something wrong.  My apologies.