Low level test for network packet loss: Guidance Needed...

Help and support for HDHomeRun DVR and HDHomeRun software for Windows 10, Mac, Android, XBox, etc.
HDHRBevo
Posts: 105
Joined: Thu Apr 16, 2009 9:40 am

Low level test for network packet loss: Guidance Needed...

Postby HDHRBevo » Fri Dec 28, 2018 2:44 pm

I am running the Low level test for network packet loss (viewtopic.php?f=15&t=5877) and need to better understand the diagnostic output. I see the guidance posted by JasonL; "You should see a series of dots. "n" indicates network packet loss. "t" indicates a reception error. "s" is informational.", but is there any other information available?

For example, I am seeing multiple "S" on one of my four tuners (none on the others). Any place to see what the "informational" S means? Also, I received one t on one tuner. Does anyone know how detrimental the t's would be to the picture. For example, would one t cause the video playback to freeze? Does this diagnostic produce a log also? If so, where would I find it?

Thanks
Gary
Last edited by HDHRBevo on Sun Dec 30, 2018 9:36 am, edited 1 time in total.

demonrik
Posts: 991
Joined: Mon May 04, 2015 10:03 am
Device ID: 10736454, 1073A35A, 1075C377

Re: Low level test for network packet loss: Guidance Needed...

Postby demonrik » Sun Dec 30, 2018 9:12 am

don't worry about S... that's just the devices communicating between themselves.
You don't want 'n' or 't', these represent data loss.
Not that they would result in a freeze - but certainly could be indicative of issues and a bug somewhere in the overall software stack could result in a freeze.

To put things in perspective..
(as I read your other thread I believe your asking for better overview of why SD is asking for all this info - ignore if this is too obvious though)
Overall player software is effectively about data movement from a packet buffer (your network) to a frame buffer (input to the video decoder) to a decoded buffer (output of the video decoder) and finally to a display buffer (which gets pushed to your TV/monitor).
A video codec will encode the video to a series of video frames varying in size depending on temporal configuration with references to previous (and sometimes future) video frames.. These encoded frames can be quite large so can span multiple network packets, and thus SD software will need to gather up a number of packets from the packet buffer to the frame buffer before submitting to the decoder to process.
If you have a reception error - then you will likely not get packets until it resolves, resulting in 'holes' in the frame buffer to the decoder. Packet loss can be either the packet is dropped by a device in the chain from transmitter to receiver or is delayed such that it is received out of order or later than required and will result in the same thing - holes in the frame buffer.
Once SD software submits the frame buffer to the decoder (could be software in some cases, but most likely hardware accelerated on your Gfx card or playback device like fireTV) it (the decoder) needs to handle those holes and respond to SD software that the frame decoded. Depending on decoder and SD software there is plenty of opportunity for the decoder to crash based on the 'hole' depending on where/when it occurs.
Note - this is a simplified version of the overall process, and on some devices there are additional stages for deinterlacing, etc., and thus even more opportunities for crashes/freezes.

So with all that said - the key to why they are asking you to run the tool is to determine if you are seeing a lot of issues where you are experiencing packet reception loss on the client, either from packet drop 'n' or reception issues 't'. Any occurrence of these results in the 'hole' in the frame buffer and potential issues. An 'n' could be something else in your network causing a switch/router in your network to drop the packet from the tuner.. a 't' indicates a problem from your service to the tuner. If your not seeing many of these, then is likely your path from tuner to device your measuring at is good.. So it could be issue from tuner to DVR or maybe DVR to client (different ports on switch is different path ;))

You don't mention which PC you are measuring - the DVR or the Client. If you're not seeing much loss from tuner to either, and you are seeing the issue mostly when skipping playback while recording the same event, the next thing to look for is if there is something happening on the DVR PC when you are reading/writing to the same file on the harddisk via the same NIC and if that's causing issues with the DVR getting packets to the NIC for the client to consume.. Hence the asks on other thread on CPU utilization spikes, etc., to determine why you are losing packets for such a use when others are not.. To be clear - there is a lot between the tuner and SDs player that can go wrong that is outside SDs control (or ability to simply build - equipment age, wear and tear, etc) - that's why all the requests from them - they are trying to get clarity on what is causing the issue so they can then reproduce, root cause and fix.

HDHRBevo
Posts: 105
Joined: Thu Apr 16, 2009 9:40 am

Re: Low level test for network packet loss: Guidance Needed...

Postby HDHRBevo » Sun Dec 30, 2018 12:44 pm

Demonrik, thanks much for you very detailed insight. Your feedback is greatly appreciated.
(as I read your other thread I believe your asking for better overview of why SD is asking for all this info - ignore if this is too obvious though)
You are correct, it is related to the problem determination tasks in that thread, but I have actually given up on getting SD to answer my questions related to their information requests and am instead trying to provide all the information I can, including, but not limited to, that being asked for.
You don't mention which PC you are measuring - the DVR or the Client.
All my tests are now done on my W10 HTPC, which acts as the DVR & (Universal) client. The fact that I can also reproduce the freezes on multiple Fire TVs, and another W10 PC using the HDHR client interface(s) is at this point, is beside the issue.

As detailed in the referenced thread, each week I have made changes in my hardware/network infrastructure to try to eliminate suspect components. For example, this week I have changed out a cat 6 cable, adjusted security software settings, and the DVR will be recording to a different internal hard disk.
if you are seeing a lot of issues where you are experiencing packet reception loss on the client, either from packet drop 'n' or reception issues 't'. Any occurrence of these results in the 'hole' in the frame buffer and potential issues. An 'n' could be something else in your network causing a switch/router in your network to drop the packet from the tuner.. a 't' indicates a problem from your service to the tuner. If you’re not seeing many of these, then is likely your path from tuner to device your measuring at is good. So it could be issue from tuner to DVR or maybe DVR to client (different ports on switch is different path
In diagnostic tests, 2 hours one day, and 6 hours the next, on all 4 tuners, I had one tuner that had 1 t the first day and a different tuner (same channel) had 2 t’s the next day. Two different tuners had multiple s’s on the same channel (different days). None of the tests showed any n’s, on any tuner. These tests were after this week’s infrastructure changes, so I will better know after today's games if THOSE changes point to THE (alleged) problem.
To be clear - there is a lot between the tuner and SDs player that can go wrong that is outside SDs control (or ability to simply build - equipment age, wear and tear, etc) - that's why all the requests from them - they are trying to get clarity on what is causing the issue so they can then reproduce, root cause and fix.
I agree and understand their challenge. Further their challenge is magnified by attempting to be all things to all devices. They simply have little outside hardware/software control over the devices that cohabit the DVR and/or clients. The T-Vo/Recast (etc.) integrated DVR model avoids some of those pitfalls, and in most cases are much less expensive. However, as for an effort to “reproduce, root cause and fix” in this case, their support is at best very SLOW to respond and at worst totally indifferent. In fact, the referenced thread, that SD forum support started, has not had a SD forum response since Dec 12th, and SD technical support has not responded at all to the ticket I opened Oct 7, 2018.
and on some devices there are additional stages for deinterlacing, etc., and thus even more opportunities for crashes/freezes.
I agree, the business SD has chosen to be in is complex. However, other software providers, even on/in my HW/SW environment (i.e. WMC), have found ways to address this complexity such that skips (or packet loss) do not freeze the picture.

Thanks again
Gary

demonrik
Posts: 991
Joined: Mon May 04, 2015 10:03 am
Device ID: 10736454, 1073A35A, 1075C377

Re: Low level test for network packet loss: Guidance Needed...

Postby demonrik » Sun Dec 30, 2018 2:29 pm

Demonrik, thanks much for you very detailed insight. Your feedback is greatly appreciated.
Just trying to help :)
All my tests are now done on my W10 HTPC, which acts as the DVR & (Universal) client. The fact that I can also reproduce the freezes on multiple Fire TVs, and another W10 PC using the HDHR client interface(s) is at this point, is beside the issue.
Would seem to be.. TBH - I don't think the low level diagnostics are going to help.
From the thread it seemed to be indicating an issue from DVR to client is causing the freeze, even if client is the same PC. Because you use loopback from DVR engine to Client it does mean it goes through the networking stack of W10 the same as if the client was out on another PC anyway.. Only difference is local client doesn't push the data to the Marvell driver, but instead to the loopback driver. So whatever is happening is very likely before that step on the DVR PC.
I think SD just getting you to verify that it's not an upstream issue from the DVR
In diagnostic tests, 2 hours one day, and 6 hours the next, on all 4 tuners, I had one tuner that had 1 t the first day and a different tuner (same channel) had 2 t’s the next day. Two different tuners had multiple s’s on the same channel (different days). None of the tests showed any n’s, on any tuner. These tests were after this week’s infrastructure changes, so I will better know after today's games if THOSE changes point to THE (alleged) problem.
IMHO I don't think this is going to show the real cause of your issue. But is just a hunch
I agree, the business SD has chosen to be in is complex. However, other software providers, even on/in my HW/SW environment (i.e. WMC), have found ways to address this complexity such that skips (or packet loss) do not freeze the picture.
Ideally it shouldn't freeze. Without knowing more data from SD it's hard to know why.. They mention data loss, but could be corruption or actual loss. They would really need some log to tell them exactly what state the app was in when it froze. could be a timeout and a bug that freezes instead of doing something else.. I notice they haven't responded since the last request to turn on all debug options, so it's really speculation on our part since we don't know whats causing it. As for others not having similar issues, they might have issues in other areas that were never exposed to because of how things were done then, versus how to do things now.

But back to your DVR PC..
You didn't mention chipset so going to assume P35 since your CPU is a Core 2 Quad and the fact you have Marvell NIC and not Intel (included in the ICH10 which was paired with P45)
Have you tried doing intensive ram test on that PC?
Going back to my point above with loopback your effectively going to be doing lots of copies through your system memory which is now probably close to 10yrs old (as is CPU and chipset). because of many copies, there could be corruption in memory causing retries or other issues that just eventually result in that data loss (silicon does degrade over time). Because you don't need to go to the Marvell in the loopback mode it's the only logical explanation to me, i.e. the CPU has to copy in data from the chipset memory controller to it's cache, then push it out again to the new location in memory for every copy. If there are issues between cpu to chipset to memory you could have something choking the data throughput. Even more so with USB or PCI devices interrupting CPU and again forcing it back and forth over the front side bus (FSB) to the read/write memory - so check for errant devices causing lots of interrupts.

Anyway - wish you luck on getting to the bottom of it all. Might be time to consider writing the PC off as dying....


Return to “HDHomeRun Software Setup & Troubleshooting (Live & DVR)”

Who is online

Users browsing this forum: No registered users and 4 guests