SERVIO+PRIME recording corruption/stuttering?

Help and support for HDHomeRun DVR and HDHomeRun software for Windows 10, Mac, Android, XBox, etc.
HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Thu Aug 08, 2019 11:44 pm

Can you please troubleshoot and help me understand what's
going on?
Oh, and I enabled diagnostics on all three
parts … SERVIO, PRIME, and XBOX One app, repro'd, and did
a channel scan.

Background:
I used to have a PC storing the recordings. It was good.
I was SUPER excited about the SERVIO … getting rid of that old PC!
SERVIO is advertised as being able to record multiple streams, and
playback multiple streams, all at the same time. I was (will) even
considering buying a second. But, they have to record reliably...

Repro Date/Time:
2019-08-08 at approximately 8:30pm?

Issue:
Stuttering in recording, playback.

Steps to Repro:
1. Set to record Seahawks game in HD (7pm-10pm, PST)
2. Start watching at 8pm
(So, playback starts 1 hour time-shifted)
3. Skip commercials & some "dead time" between plays,
until nearly caught up to real-time playback.
Commercial skips are 8x-10x skips forward of ~30s each.
"Dead time" skips are 1x skip forward after each play ends.

Expected Results:
Playback remains smooth and uninterrupted.
Can watch a football game in 1 hour, instead of 3 hours.

Actual Results:
1. As get near real-time (watching being close to live),
video stutters terribly, and eventually even the recording
starts being corrupted.
2. Playback controls become unresponsive (play/pause
do nothing) at corrupt video.
3. Exiting the app and restarting playback appeared to allow
playback to resume ... for maybe 20 minutes ... before
hitting corrupt video again.
4. Eventually, playback threw some sort of media error
on the Xbox One. (didn't think to record the error,
sorry ... something about a media playback or decoder error?)
5. Some areas seem corrupted at the recording, as they
continue to show as corrupted at later playback.

Device Info:
HDHomeRun SERVIO Device ID: 15301857, FW 20190621
HDHomeRun PRIME Device ID: 13182F70, FW 20190621
Signal Strength 95% (-3.0 dBmV)
Signal Quality 100% (34.2 dB)

From EpisodesURL for NFL football:

Code: Select all

[{"Category":"sport","ChannelAffiliate":"FOX","ChannelImageURL":"http://img.hdhomerun.com/channels/US21254.png","ChannelName":"KCPQDT","ChannelNumber":"113","EndTime":1565326800,"EpisodeTitle":"Denver Broncos at Seattle Seahawks","FirstAiring":1,"ImageURL":"http://img.hdhomerun.com/titles/C191277EN186U.jpg","OriginalAirdate":1565222400,"ProgramID":"EP000031281965","RecordEndTime":1565326830,"RecordStartTime":1565315971,"Resume":8309,"SeriesID":"C191277EN186U","StartTime":1565316000,"Synopsis":"Football action from around the National Football League.","Team1":"Denver Broncos","Team2":"Seattle Seahawks","Title":"NFL Football","PlayURL":"http://192.168.10.56:80/recorded/play?id=6fda8bfd","CmdURL":"http://192.168.10.56:80/recorded/cmd?id=6fda8bfd"}]

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Fri Aug 09, 2019 2:10 am

Update:
I downloaded the recording using VLC (got URL via web APIs .. ~ 12GB).
The video has some video macroblocking at ~12m10s., but the real bad part is at 1h09m25s.
At that point, the video nearly stops, it gets so bad.
Decoding simply fails in VLC at ~1h10m11s (no video shows, playback appears to freeze).

Any attempt in VLC (playing back the local file) to skip past 1h10m11s instead seeks to ~50m30s.

Based on this, it appears that when we started playback while it was in the process of recording, the recording itself was corrupted … so it's not a problem getting it out of the SERVIO into the XBOX One, so much as it getting the stream onto the SERVIO in the first place.

leenypost
Posts: 71
Joined: Tue Aug 15, 2017 11:14 am

Re: SERVIO+PRIME recording corruption/stuttering?

Post by leenypost » Fri Aug 09, 2019 6:22 pm

[Mod edit: removed off topic FUD rant about 100 meg NIC speeds - NS]

NedS
Silicondust
Posts: 427
Joined: Mon Dec 10, 2018 12:38 pm

Re: SERVIO+PRIME recording corruption/stuttering?

Post by NedS » Fri Aug 09, 2019 7:07 pm

leenypost wrote:
Fri Aug 09, 2019 6:22 pm
[Mod edit: removed off topic FUD rant about 100 meg NIC speeds - NS]
None of that has anything to do with the issue being reported in this thread.

jasonl
Silicondust
Posts: 14233
Joined: Sun Oct 28, 2007 9:23 pm

Re: SERVIO+PRIME recording corruption/stuttering?

Post by jasonl » Sun Aug 11, 2019 3:52 pm

The logs show there was a burst of network packet loss between the PRIME and SERVIO about :10pm, and then it was fine until 8:10 when it started dropping heavily until about 8:45, and it was fine from then until just before 9 when it started again. Something isn't right at the network level. Might be a problem with a network cable (either HDHomeRun-switch or SERVIO-switch) or the switch itself. Can you try replacing the cables? Any chance you have another switch you can test with to rule out a problem with the existing one?

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Sun Aug 11, 2019 8:33 pm

jasonl wrote:
Sun Aug 11, 2019 3:52 pm
The logs show there was a burst of network packet loss between the PRIME and SERVIO about :10pm, and then it was fine until 8:10 when it started dropping heavily until about 8:45, and it was fine from then until just before 9 when it started again. Something isn't right at the network level. Might be a problem with a network cable (either HDHomeRun-switch or SERVIO-switch) or the switch itself. Can you try replacing the cables? Any chance you have another switch you can test with to rule out a problem with the existing one?
Hi Jason,

Thanks for helping. Given the recording itself was corrupt, it would make sense to be between the two.
I can move the SERVIO to be closer (network wise) to the PRIME, and replace the cables with fresh ones, so that the only network hop between them will be a Cisco SG300-10. I will then re-enable logs and … wait for the next pre-season football game, I guess. :)

BTW, Is there any way to grab the logs locally for review? I would have loved to be able to see this w/o using your time....

jasonl
Silicondust
Posts: 14233
Joined: Sun Oct 28, 2007 9:23 pm

Re: SERVIO+PRIME recording corruption/stuttering?

Post by jasonl » Mon Aug 12, 2019 2:17 pm

You can use the command line to see the live status, but there's nothing that will actually log it to a file.

"%programfiles%\silicondust\hdhomerun\hdhomerun_config.exe" 13182F70 get /tuner0/debug

Use the actual tuner that's in use (may be 0, 1, or 2). What you're looking for is the err number on the net: line. This indicates the number of packets lost on the network side, usually due to network problems.

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Fri Aug 30, 2019 12:41 am

Thanks! that's a useful item for self-troubleshooting … using it with "get help" is recommended for others who also want a bit more cmd-line logging goodness.

I have not had problems with corruption/packet loss since moving the servio into the wiring closet with the prime. So, that is great news.

However, the system has failed to record the last two preseason football games at all, even though other shows would record fine. This is making me VERY unpopular with the wife, who is actually a bigger football fan than I am. As the regular season is about to start, I need to be 100% sure it will record, or risk sad / angry emotions here … any help offered would be appreciated.

It's possible this is related in part due to some UI … idiosyncrasies … when trying to setup football to record a single team. I went in and deleted the task claimed to be recording the NFL for our single team, using the Win10 app. I had to delete about four such tasks. Previously, my wife couldn't get the task to record "1h After" the scheduled end. I will open a different thread on these UI bugs, but...
  1. Is it expected that there could be multiple tasks trying to record the same NFL game?
  • If one or more tasks were set to record the game 30s early, and at least one other task set to record starting 5 minutes early, could that have caused the recordings to start, only to fail 270 seconds later?
Some apparently relevant portions of the prime's system logs:
20190830-01:59:29 Tuner: tuner0 tuning 1013 KCPQDT (auto:231MHz-1)
20190830-01:59:29 Tuner: tuner0 streaming http to 192.168.10.56:33145
20190830-01:59:29 CableCARD: tuner0 1013 KCPQDT (auto:231MHz-1) access = subscribed
[[ … entries omitted … none related to tuner0, none related to channel 1013 … ]]
20190830-05:58:36 HTTP: rejecting request from 192.168.10.56 - all tuners in use
20190830-05:58:37 HTTP: rejecting request from 192.168.10.60 - all tuners in use
20190830-05:58:37 HTTP: rejecting request from 192.168.10.60 - all tuners in use
20190830-06:00:00 Tuner: tuner0 http stream ended (remote closed)
20190830-06:00:25 Tuner: tuner0 tuning 109 KCTSDT (auto:159MHz-1)
20190830-06:00:26 Tuner: tuner0 streaming http to 192.168.10.56:33167

This appears to show that the prime got a request at 7pm local time (2am next day UTC), and successful streamed channel 1013 for four hours … exactly matching the intended 3 hour timeslot + "1h After" setting we thought we had set for the football game.

Unfortunately, no game appears to have recorded....

Ideas? I need this to reliably record the stuff it's saying it will record... I'm a kickstarted supporter, and one who wants to champion this product. Help!?

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Fri Aug 30, 2019 12:58 am

Is my Servio defective? Gut check says that having 50% of all raw reads require error correction, and even if every read required a seek, that's nearly ~2% seek errors... these error rates seem a bit high (even with the understanding that this is an AV/Video targeted drive, and even though it's using SMR):

Code: Select all

436,753,997 Total_LBAs_Read
204,082,736 Raw_Read_Error_Rate
  3,848,525 Seek_Error_Rate
Full smart output:

Code: Select all

C:\>"%programfiles%\silicondust\hdhomerun\hdhomerun_config.exe" %servio_id% get /hdd/smart
smart: pass
ID# NAME                    RAW             VAL WST THR
  1 Raw_Read_Error_Rate     204082736        83  64   6
  3 Spin_Up_Time            0               100 100   0
  4 Start_Stop_Count        3               100 100  20
  5 Reallocated_Sector_Ct   0               100 100  10
  7 Seek_Error_Rate         3848525          66  60  45
  9 Power_On_Hours          1064             99  99   0
 10 Spin_Retry_Count        0               100 100  97
 12 Power_Cycle_Count       3               100 100  20
184 End-to-End_Error        0               100 100  99
187 Reported_Uncorrect      0               100 100   0
188 Command_Timeout         0               100 100   0
189 High_Fly_Writes         0               100 100   0
190 Airflow_Temperature_Cel 35               65  57  40
191 G-Sense_Error_Rate      0               100 100   0
192 Power-Off_Retract_Count 2               100 100   0
193 Load_Cycle_Count        47              100 100   0
194 Temperature_Celsius     35               35  43   0
197 Current_Pending_Sector  0               100 100   0
198 Offline_Uncorrectable   0               100 100   0
199 UDMA_CRC_Error_Count    0               200 200   0
240 Head_Flying_Hours       1064            100 253   0
241 Total_LBAs_Written      1964595752      100 253   0
242 Total_LBAs_Read         436753997       100 253   0
254 Free_Fall_Sensor        0               100 100   0

NedS
Silicondust
Posts: 427
Joined: Mon Dec 10, 2018 12:38 pm

Re: SERVIO+PRIME recording corruption/stuttering?

Post by NedS » Fri Aug 30, 2019 3:40 pm

HenryGab wrote:
Fri Aug 30, 2019 12:58 am
Is my Servio defective? Gut check says that having 50% of all raw reads require error correction, and even if every read required a seek, that's nearly ~2% seek errors... these error rates seem a bit high (even with the understanding that this is an AV/Video targeted drive, and even though it's using SMR):

Code: Select all

436,753,997 Total_LBAs_Read
204,082,736 Raw_Read_Error_Rate
  3,848,525 Seek_Error_Rate
Full smart output:

Code: Select all

C:\>"%programfiles%\silicondust\hdhomerun\hdhomerun_config.exe" %servio_id% get /hdd/smart
smart: pass
ID# NAME                    RAW             VAL WST THR
  1 Raw_Read_Error_Rate     204082736        83  64   6
  3 Spin_Up_Time            0               100 100   0
  4 Start_Stop_Count        3               100 100  20
  5 Reallocated_Sector_Ct   0               100 100  10
  7 Seek_Error_Rate         3848525          66  60  45
  9 Power_On_Hours          1064             99  99   0
 10 Spin_Retry_Count        0               100 100  97
 12 Power_Cycle_Count       3               100 100  20
184 End-to-End_Error        0               100 100  99
187 Reported_Uncorrect      0               100 100   0
188 Command_Timeout         0               100 100   0
189 High_Fly_Writes         0               100 100   0
190 Airflow_Temperature_Cel 35               65  57  40
191 G-Sense_Error_Rate      0               100 100   0
192 Power-Off_Retract_Count 2               100 100   0
193 Load_Cycle_Count        47              100 100   0
194 Temperature_Celsius     35               35  43   0
197 Current_Pending_Sector  0               100 100   0
198 Offline_Uncorrectable   0               100 100   0
199 UDMA_CRC_Error_Count    0               200 200   0
240 Head_Flying_Hours       1064            100 253   0
241 Total_LBAs_Written      1964595752      100 253   0
242 Total_LBAs_Read         436753997       100 253   0
254 Free_Fall_Sensor        0               100 100   0
I'm told that these values are normal for the SMART test.

nickk
Silicondust
Posts: 15548
Joined: Tue Jan 13, 2004 9:39 am

Re: SERVIO+PRIME recording corruption/stuttering?

Post by nickk » Fri Aug 30, 2019 5:14 pm

Pretty sure the Raw_Read_Error_Rate is a multi-part number, not an integer.

The drive is quite happy.

Nick

jasonl
Silicondust
Posts: 14233
Joined: Sun Oct 28, 2007 9:23 pm

Re: SERVIO+PRIME recording corruption/stuttering?

Post by jasonl » Sat Aug 31, 2019 4:15 pm

The RAW numbers for Raw_Read_Error_Rate and Seek_Error_Rate are proprietary to the manufacturer and are not counters. There's some formula that translates that number into something meaningful, but only WD knows the formula for WD drives, and only Seagate knows the number for Seagate drives, and so on. For those, you have to look at the VAL(ue) and THR(eshold). They're on an inverted scale, so if your value goes below the threshold, you have a problem. Yours aren't, so your drive is fine based on the manufacturer's guidelines.

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Sat Aug 31, 2019 6:51 pm

OK, I can accept that the raw read error number is a proprietary value. Glad to know the threshold is inverted … My thanks to both of you for your insight. Unfortunately, that means the problem of NFL not recording is not related to this.... so something else is awry.

Is there anything I can do ahead-of-time to help track why NFL games don't appear to have been recorded?

The first regular season game is coming up, and I can likely get a single-game pass from the wife, so long as I can give a straight-faced answer that we're getting closer to a fix....

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Sun Sep 01, 2019 4:18 pm

JasonL, NickK, NedS:
THANK YOU for each of your help figuring out the multiple root causes!

UPDATE:
The Servio actually DID record the two prior pre-season games. However, it never listed those games in the UI (checked with XBOX One, iPhone, and Win10 clients) until I rebooted the Servio. Now, the two prior recorded games have appeared, and my wife is actively "testing" the results.

I have filed separate threads on the bugs: Accordingly, I think we can close this thread as resolved, with cause of corrupted recordings being "customer network environment causing excessive packet loss".

HenryGab
Posts: 18
Joined: Sat Feb 16, 2008 4:28 pm
Device ID: 15301857 13182F70

Re: SERVIO+PRIME recording corruption/stuttering?

Post by HenryGab » Mon Sep 16, 2019 1:21 pm

I really don't like latent bugs. The excessive packet loss diagnosed above appears to have caused episodes to not appear until a reboot. This suggests that excessive packet loss may be able to cause one or both of the following:
  • In-memory corruption, perhaps related to (cached?) file system structures associated with the series
  • If file system is checked/verified on reboot, corruption of on-disk structures associated with the series
Either one of these could obviously be very bad. Without knowing the exact type of bug, the full extent of the problem (exploitability, frequency, etc.) cannot be known. In particular, if in-memory corruption, and the value or address can be attacker-controlled, … could be bad. I want this product to be rock-solid, so please accept my thoughts on something that would have a one-time setup cost, but might enable a local repro for true root-cause analysis....

Would it be reasonable to ask that a long-term repro like this is attempted?

Especially if you have the hardware to programmatically enable network packet loss, something like the following:
  1. Setup various series to record all episodes.
  2. Configure network to create pseudo-random(*) network packet loss, with various loss rates and distributions during each recording
  3. During each recording, validate that the currently-recording episode appears for playback
    1. if not, PRIOR recording from the same series may have caused file system corruption!
  4. After each recording, validate file system structures and that the just-recorded episode appears for playback
    1. if not, THIS recording may have caused file system corruption!
  5. Once there are two shows successfully recorded, delete them (in case only occurs when packet loss in first recording of a series)
Although this is just my best guess without real debugging information, the above matches the two test points I would look for the repro at.

(*) The logs diagnosed above, which indicated high packet loss, may be useful to target packet error rates, vs. just random selection.

Post Reply