QNAP QPKG for DVR Record Engine

[demonrik]

well caught..

There was another one in there which was the remove logfile. Was partially done.
I missed this because I always had same logfile name in new and old dir, so it just worked.. Should really have paid attention to the content.

Anyway - thanks for finding.. and fix is acceptable

New Package q0.8.2 is now up on the google shared drive https://drive.google.com/folderview?id= ... sp=sharing

[avdp]

working great... including the delete that I've never tried before. Thanks!

Another small concern. It seems the initial load of the page is very very slow. Something like 90 seconds for the page to show up. I don't recall it being that slow before the rules functionality was implemented, but I certainly could be wrong about that. Is it just my system, or have you noticed the same?

[avdp]

I found the problem, but not a solution yet.

in dvrui_rules.php, the file_get_contents call take a very long time - 1-2 minutes.
Which is weird, because the same URL (auth string and all) take 2 seconds with curl on the same nas, and milliseconds from chrome on my PC.

So I've been experimenting with setting http timeout headers, and setting socket timeouts, etc. But no luck so far. I can get it to timeout, but no content is returned. Very strange.

Code: Select all

$rules_json = file_get_contents($this->recordingsURL . $auth);

[demonrik]

I found the problem, but not a solution yet.

in dvrui_rules.php, the file_get_contents call take a very long time - 1-2 minutes.
Which is weird, because the same URL (auth string and all) take 2 seconds with curl on the same nas, and milliseconds from chrome on my PC.

So I've been experimenting with setting http timeout headers, and setting socket timeouts, etc. But no luck so far. I can get it to timeout, but no content is returned. Very strange.

Code: Select all

$rules_json = file_get_contents($this->recordingsURL . $auth);

Could be a bunch of things.
That function can cause issues with some servers, but am not seeing issue here.
I know there are some considerations on certain characters of the auth keys, but not seen it yet.
Could be a DNS issue too.
Or even just the amount of rules.

Could you try the dump_rules.py from my hdhr2plex project?
Or try construct the url manually and do a wget on it?

[demonrik]

@avdp

I've done a build with a few changes up on my google drive that I wonder if you could try out.
is HDHomeRunDVR_q0.8.2-a1.qpkg

This build changes the UI such that Rules and logfile list are scanned on opening the tab - not on loading the page, effectively pushing down the loading until you need it, so should speed up the loading of the page.

also in includes/dvrui_rules.php I've done a few things.
Firstly I've added an execution time around the call to the URL for rules download. This is the first line that then gets dumped to the rules tab along with the rules.. It's in milliseconds
I've also added a second method on loading the JSON for the rules.
You can comment out either method in the file and chose the one you want to try and get the execution time.

For me method 2 is on average about 10ms slower than method 1..
Curious to see what your times are.

[avdp]

I didn't realize this was your tool as well! I use it too, although not for plex - I just use it to create symlinks for ****. I did end up changing the code significantly to meet my needs.

Anyway, back to the QNAP package...

dump_rule.py also takes a while (1-2 minutes)....

[demonrik]

I didn't realize this was your tool as well! I use it too, although not for plex - I just use it to create symlinks for ****. I did end up changing the code significantly to meet my needs.

Anyway, back to the QNAP package...

dump_rule.py also takes a while (1-2 minutes)....

lol..
well feedback, or pull requests are always welcome

If dump_rule.py is also taking a long time.. I wonder if it's just the amount of rules..
How many rules do you have? (it's actually the largest priority number in the dump)

[demonrik]

oh, and some other things to try...

Maybe it's a DNS issue.
[s]if you run

Code: Select all

nslookup my.hdhomerun.com

Then try replacing the name of the server with the IP in the dvrui_rules.php and see if that speeds things up.
If that runs quickly - you may need to check your DNS settings[/color][/i][/s]
EDIT: just tried - doesn't work. seems SD have a virtual server running so needs the server address in the http request

also try doing a tracert to the my.hdhomerun.com server and see if something jumps out as an issue.

[avdp]

ok, so I installed HDHomeRunDVR_q0.8.2-a1.

Initial page load is much faster, probably because (from what I guess without looking at the code) you're no longer preloading the rules. Good, that's a great idea regardless of my particular problem.

Clicking on the rules tabs, it still takes 1-2 minutes for anything to show up. 137.52257514 is the number that shows on top (I assume you put some timer in there?). I have 38 rules, not small but not great.

I don't think it's a coding issue obviously, I also don't think it's a DNS problem since retrieving the json directly from wget on the same server is quick. But I do think it's something specific to my nas, so I am gonna keep trying a few things to sort it out.

[demonrik]

ok, so I installed HDHomeRunDVR_q0.8.2-a1.

Initial page load is much faster, probably because (from what I guess without looking at the code) you're no longer preloading the rules. Good, that's a great idea regardless of my particular problem.

Clicking on the rules tabs, it still takes 1-2 minutes for anything to show up. 137.52257514 is the number that shows on top (I assume you put some timer in there?). I have 38 rules, not small but not great.

I don't think it's a coding issue obviously, I also don't think it's a DNS problem since retrieving the json directly from wget on the same server is quick. But I do think it's something specific to my nas, so I am gonna keep trying a few things to sort it out.

I have one more method to try out. Will try and get time to implement later
But does seem related to your system.
And my bad... Time is seconds, so your seeing 2 mins to resolve 1/2 the rules I get in 200ms

[avdp]

on hdhr2plex...

My issue with it is that matching by air date just didn't yield great results for me. Perhaps the quality of the guide data where I am is not great. So I needed it to either trust the filename for season/episode, use the air date, or in some case match on episode name... I also had issues with the show name not being found, because the guide had a show name that was just too different from tvdb.

So I made it configurable on a show by show basis. I added a shows.conf file where I can specify to match on Air date (A), episode name (N), or recorded date (R). Also, for a given show, I could overwrite the seriesname to do the tvdb lookup.

Things like the news worked best based on the record time (they all had the same air date in the '70s). Things like Jeopardy worked great with the air date. Other things worked best on episode name.

For episode name matching, I used the fuzzywuzzy library to score the string compare and use the best score.

BUT since I didn't know much about python, I figured out what your code does by deconstructing it (removing stuff I didn't need, lots of trial and error, etc) before I started to add stuff I needed back in. So bottom line, to merge any of this new functionality back into your excellent program is gonna be some work. If you have any interest in the functionality I describe, I promise I will find time to incorporate it back into your code. Or I can send you my python script and see if there is anything you like out of it. Either way.

By the way - even with hdhr2plex, the tvdb lookups were taking forever. I ended up "fixing that" by adding a socket.setdefaulttimeout(5) on top of the script and it worked fine after that. I will try something similar with the dump util to see if that help. There is something funky with networking on my nas and/or network, but I can't quite figure out what it is yet.

Example entries in my shows.conf

Code: Select all

[The Voice]
[NBC Nightly News with Lester Holt]
matchon: R
[Jeopardy!]
matchon: A
seriesname: Jeopardy!
[America's Test Kitchen from Cook's Illustrated]
matchon: N
seriesname: America's Test Kitchen

No matter what, I thank you again for writing these utilities. It is thanks to these utilities that I now rely 100% on hdhomerun dvr for all my recordings!

[avdp]

yep, just as I thought... dump_rules.py is fast if I add the follow on top of the script...
So I don't know why the socket doesn't close on its own. weird. weird.

Code: Select all

import socket
socket.setdefaulttimeout(5)

By the way, the dump_rules.py shows an errors after dumping the json on the screen.

Code: Select all

Traceback (most recent call last):
  File "dump_rules.py", line 96, in <module>
    discoverHDHRList()
  File "dump_rules.py", line 88, in discoverHDHRList
    ' Title: ' + str(recording['Title'])
KeyError: 'Priority'

[demonrik]

thanks for the comments.
On hdhr2plex - I would definitely be interested in viewing your changes, see what if anything I can pull in.
I was planning to do some changes with the hdhr2plex scripts soon and to build up lists for full delete, delete but rerecord, etc..
So I can probably pull in more info on the matching.. Just not had time to concentrate on it recently
So please - PM me, or join the discord channel https://discord.gg/0uzKaTiMjCwvanxM and post the info for me to pick up

As for the timeout...
try replacing method 2 in dvrui_rules.php with

Code: Select all

		//method 2
		$context = stream_context_create(
				array('http' => array(
					'header'=>'Connection: close\r\n',
					'method'  => 'POST',
					'content' => '',
					'timeout' => .5)));
		$rules_json = file_get_contents($this->recordingsURL . $auth,false,$context);		

this even made an impact for me - reducing processing time to 100-150ms!
so looks like a good idea to set

[demonrik]

By the way, the dump_rules.py shows an errors after dumping the json on the screen.

Code: Select all

Traceback (most recent call last):
  File "dump_rules.py", line 96, in <module>
    discoverHDHRList()
  File "dump_rules.py", line 88, in discoverHDHRList
    ' Title: ' + str(recording['Title'])
KeyError: 'Priority'

that shouldn't happen.. You should always have a Title and a Priority key - so something went wrong with the JSON you got..
As part of the dump I printout the full json response.. if you could upload and PM me the link (or send on discord) I'll take a look.

[avdp]

ok, so the latest change did return nothing (no rules) but only took 10.5826029778 seconds.
I was surprised by that!

I'll PM the dump and my modest attempt at hacking python