QNAP QPKG for DVR Record Engine

[demonrik]

Really great work here. I appreciate everything you have done!

You're welcome!

Having said that, just curious as to what the end game is here. How far do we go with an app like this? All the talk of finding a show on a webpage and having it automatically record seems like it would be way out of the scope of what is even really plausible with the system. Especially considering the somewhat limited EPG that SD provides, and also it seems something more suited for a browser plugin than an app for a NAS.

My End game is to provides something to manage the DVR from the Engine perspective, i.e. on the NAS. That for me includes
- configuration
- log file viewing
- rules management
- recordings management
- upcoming/conflict management

So, we're nearly there in terms of big features. Now tweaking and refinement needed.

One last little criticism... Skewmorphism is dead. Could we get some more flat looking buttons on the server tab?

I honestly am terrible at web page design. I know what you mean though, and do want to provide something more in keeping with the other tabs. Might play around with that once I get some of the other bits working

deamonrik, you are flat-out crazy I can't believe the product you are providing us; this is what I expect from SiliconDust and you are well ahead of the curve. You have made it extremely easy to implement the DVR Record engine on the Qnap NAS, and thus are helping us all enjoy the test phase that much more. Kudos!!! I hope SD is compensating you somehow with some free tuners or something!

You're welcome.. no compensation, just a vested interest in doing what I'm doing..

Quick comment...if a show has a two-line description on the Recordings or Rules pages, then the green text (channel and "All Episodes") or the Buttons (play, delete, rerecord) is pushed down where its sitting half-way on the horizontal line separating the two episodes. I wonder if it's a font rendering issue on my chromebook browser vs what you are using to develop the page.

It's a combination of fonts and web page layout. It's definitely my weaker point and I'm hoping to spend a bit more time to get it all to scale properly.

The Upcoming page looks fine, because the thumbnail images are larger, giving more room for the text and the green buttons/text.

Which is something I'm considering for the others... BUT I want something that scales properly just in case there is something that forces an increase again. So before I do that for all the pages, I need to get the scaling properly working.. It shouldn't overlap like that.

[signcarver]

Those were my thoughts exactly and I originally wanted to do the same on the simple My Cloud but when they changed the page size, it became tough to ask one to install some needed server components for a real LAMP server so I abandoned my dream of doing something similar on it (though did have it on a ReadyNAS... trying to get a QNAP to play with demonrik's package)

I only use my web page to schedule... makes it easy to grab a phone or tablet to search and schedule, and my behavior for this started before one could tell kodi to start in DVR rather than tie up a tuner... I don't "ping" the engine so it might not pick it up quickly (when the android app was just released it behaved like this but was fixed within a day)... I don't know what "pinging" the engine really means, perhaps just getting the json for discover or recordings but I never really looked into it.

I'll briefly show a picture of one of my older rules and search pages (hadn't uses imgur before hopefully it worked for others)



At that point the quick record button added a default set padding (controlled from admin page and set as cookie so would be there next time same device visited) and I had just took out the option for recent/always on quick record as typically 90% of search results would not be eligible for recent so I abandoned that idea (but could easily add it back as a default, I'd rather do something "smart" and check episodes first and see if recent can apply, otherwise ignore it for quick record), the idea behind episodes was to schedule a one time recording... also since then I added more pull downs for padding and date selection and such)... I also toyed with a version that the drop down on rules was actually a multiple selection box and would allow a faster way to adjust each padding (many at once), deleting many, and set AOAO to something... I am more of a back end developer (when it comes to developing) so a UI isn't really my thing but my implementation works quite well for me... it would not surprise me if SD did something similar (probably same rui though) from the my.hdhomerun.com page (or even have the engine serve such a page when you click on it) but there are basically two ways to do it (client side... everything is in javascript and can be hosted anywhere but you have to access from within your network to pull the current DeviceAuths... or server side where the server makes the requests but can be accessed from anywhere. I just feel that when in front of a computer looking at a web page there is more room to offer other attributes when one doesn't require the use of a dpad that nearly anything is possible.

These little simple things is why I love the project so much as it can be highly customized to an individuals preferences and never really faulted SD for anything that was not yet developed. Even when Channels made its "demands" of missing features, I felt most were already implemented but quickly understood that due to the live nature of Channels, they needed a bit better such as actual start and end time, not when the recording was supposed to start or end as one might be watching a show and click record starting 10 minutes in... in my world rules are set at least a few days before so it is rarely an issue.

I also have planned a way to filter results by favorites and even a way that when one has multiple devices to filter by multiple favorite lists and give them friendlier names (such as Kids Room instead of device ID) This would allow for some (multiple) favorites lists to always be present, in addition to ones that are defined. Even though you see a guide button there, right now it does nothing but that is where I think these filters would perform best (if I could use "suggestions" similar to RUI I'd like such filters there too... I'd also like to filter on SD's "genre" filters as well) but I have no doubt these things will come over time, if not by SD by someone else.

As I have mentioned before I am paranoid about sharing such things and violating ToS (my images might even have as I probably should have blurred images and descriptions) so right now I am waiting a little bit longer before I share anything "real" (I might provide a hint on Search to someone)

And sorry if I am derailing this thread a little

[demonrik]

Those were my thoughts exactly and I originally wanted to do the same on the simple My Cloud but when they changed the page size, it became tough to ask one to install some needed server components for a real LAMP server so I abandoned my dream of doing something similar on it (though did have it on a ReadyNAS... trying to get a QNAP to play with demonrik's package)

You could just take the UI off the github and simply reconfigure the vars.php for your local system. At this time I think you only need to modify the following vars

Code: Select all

	const DVR_qpkgPath	="/share/CACHEDEV1_DATA/.qpkg/HDHomeRunDVR";
	const DVR_bin		="hdhomerun_record_linux";
	const DVR_sh		="HDHomeRunDVR.sh";
	const DVR_config	="HDHomeRunDVR.conf";

The recordings folder, etc.. is all dynamically discovered once the UI can find the CONF file. The .sh is my startup script for the engine - again, something that can be reused on any linux system.

I'll briefly show a picture of one of my older rules and search pages (hadn't uses imgur before hopefully it worked for others)

I'd be interested in knowing more about the search method.

I am more of a back end developer (when it comes to developing) so a UI isn't really my thing but my implementation works quite well for me...

Same here - but can be fun making things look intereiitng

it would not surprise me if SD did something similar (probably same rui though) from the my.hdhomerun.com page (or even have the engine serve such a page when you click on it) but there are basically two ways to do it (client side... everything is in javascript and can be hosted anywhere but you have to access from within your network to pull the current DeviceAuths... or server side where the server makes the requests but can be accessed from anywhere. I just feel that when in front of a computer looking at a web page there is more room to offer other attributes when one doesn't require the use of a dpad that nearly anything is possible.

I hope they do add something like this.. Yeah, then seems like I wasted quite a few hours, but a lot of what could be done would be better served to do back in their servers rather than pulling teeth through the current API set.

As I have mentioned before I am paranoid about sharing such things and violating ToS (my images might even have as I probably should have blurred images and descriptions) so right now I am waiting a little bit longer before I share anything "real" (I might provide a hint on Search to someone)

I hear you.
I stick to what was in that post by nickk.. I don't store the images or cache them.
It could be argued that posting a screenshot is archiving, but I would argue that it's ok under fair use as it's not about showing the image of a show for the show sake, but the UI being developed.. That said if SD or another requests I remove the screenshot I will do so..

And sorry if I am derailing this thread a little

Not at all.. this thread has been nicely evolving.. Your search has peaked my interest though. I'm guessing it's a text search and then once found you have the SeriesID and use that to then get the Episode list. I've not seen any docs on it, nor seen you mention it elsewhere.. Understand totally if you don't want to expose it without SDs blessing. I even waited a couple of months after you exposed the others to see if SD removed the posts - they didn't so I assume its ok..

On another note.
I was looking into trying to optimize the 'upcoming' page but it just cannot work.
The only way to optimize is to reduce the number of requests for episodes off the seriesIDs from the rules.
I was thinking I could use the scroll to load up more upcoming episodes, but that can't work.. Because even if I was to limit the episode requests to a few rules, it takes just one out of order to push an episode to the front of the scroll and the whole effect is ruined. I could of course just not sort based on start time... might consider that in the far future.
TBH it would be just so much nicer if the API for episodes allowed me to specify a time range.. i.e. give me all the episodes that have a recording rule set with a start time of > X
But that doesn't exist.. so it is what it is and I'll go look at other tweaks to be done.

[avdp]

Your search has peaked my interest though. I'm guessing it's a text search and then once found you have the SeriesID and use that to then get the Episode list. I've not seen any docs on it, nor seen you mention it elsewhere.. Understand totally if you don't want to expose it without SDs blessing.

I am working on the search capabilities, I am very close. signcarver provided enough of a "hint" in a previous post when he mentioned the HDHomeRun for Kodi. That addon is just a bunch of python scripts under GPL license. So they're fair game

[signcarver]

I'm old school and looking at code "taints" projects, so I try to avoid it except as a learning scenario... I just knew I would need a text box to type in a term and search for that term and display the results.

Here is what bothered me:

Back prior to DVR this was posted for the Kodi Add-On

The TV guide data used by the HDHomeRun Kodi add-on is licensed for use with Silicondust software when used with Silicondust products. Modifications to the HDHomeRun Kodi add-on that do not substantially change how the TV guide is used should be fine. For other uses please contact Silicondust.

Rule - no storing or archiving guide data or images.

Guide data may be cached for as long as the data is current (typically 1-2 hours).
Images may be cached for up to the cache duration specified in the HTTP header (typically 24 hours).
Images may not be stored within a recorded file.

the DVR ToS is

Guide data and/or images may not be used outside of Silicondust software without written agreement.
Please contact info@silicondust.com if you would like to write a client app that works with the HDHomeRun DVR system and makes use of Silicondust guide data and/or images.
Images must not be archived or stored. Images may be cached following the cache duration and rules in the HTTP header.

Back when I started mine I made the assumption that using it for the DVR was "for use with Silicondust software when used with Silicondust products" as the engine was the SD software, now they say "Guide data and/or images may not be used outside of Silicondust software without written agreement" as mentioned some may decide that a URL is an image (shouldn't be as long as not stored) and I am not certain what exactly Guide Data pertains to... to me at least search and probably episodes should be fair game though episodes touches more on actual Guide Data/schedule so they might not view it that way, the Guide API is most likely what should be avoided without written permission (but that is my opinion and as you mentioned the add-on should be GPL, but using the data outside of it may be off limits). Suggest API probably should be treated as proprietary to VIEW as that should be SD's baby (though I would love to filter those by favorites).

[avdp]

the Guide API is most likely what should be avoided without written permission (but that is my opinion and as you mentioned the add-on should be GPL, but using the data outside of it may be off limits).

I understand the concern. There is a rich history of ToS being incompatible with the GPL - the apple store is one of those, and this may be another. Anyway, IANAL, as long as we don't abuse the APIs, it should be of no concern for SiliconDust. Hopefully this will be clearer once they update/complete the documentation for the APIs. I am actually really enjoying the simplicity and clarity of the APIs they've exposed, they're a great foundation to fix all the shortcomings as the official client / add-on.

[moderator - removed talk of patents]

[signcarver]

Those were my thoughts exactly and I originally wanted to do the same on the simple My Cloud but when they changed the page size, it became tough to ask one to install some needed server components for a real LAMP server so I abandoned my dream of doing something similar on it (though did have it on a ReadyNAS... trying to get a QNAP to play with demonrik's package)

You could just take the UI off the github and simply reconfigure the vars.php for your local system. At this time I think you only need to modify the following vars

Code: Select all

	const DVR_qpkgPath	="/share/CACHEDEV1_DATA/.qpkg/HDHomeRunDVR";
	const DVR_bin		="hdhomerun_record_linux";
	const DVR_sh		="HDHomeRunDVR.sh";
	const DVR_config	="HDHomeRunDVR.conf";

The recordings folder, etc.. is all dynamically discovered once the UI can find the CONF file. The .sh is my startup script for the engine - again, something that can be reused on any linux system.

OK getting around to it a little this weekend... got latest from git as a zip and put UI onto a single bay my cloud... however I am testing on one not running the engine... what I had hope would have been for engine discovery (the new way from my.hdhomerun.com) and pulling in the ip:port of that to manage recordings... I'll have to take a closer look at it to see how I want to change this (I realize your main purpose at first was probably status, start, and stop the engine so it is to be expected and it is only recently that the record engine info was online and one had to set it and most don't know their port). Keep in mind that this device isn't meant to install packages on, I totally ruined php on the device one time trying to do an update for a utility I needed for a web page as it uses the wrong page size for the version it is on so everything must be recompiled.

One question since I haven't looked at it much to absorb it but are requests made from the browser using javascript
(at a glance this seems to be) or are they posted to the server to make the request and/or the server initially making the discovery and getting the DeviceAuths... the advantage of the later is that one can do things more remotely (particularly with latest addition (I won't mention that tab)) but it makes the server (in your case qnap) do all the gets from SD rather than the client browser... I also saw a sleep or two in the scripts (I'm debugging through browser so not looking at the php at this time, just making notes of what is working or not) and I think it would be preferred to use either callbacks or promises than the sleep.

So far it seems to work well for the Rules (though I would like to edit there and will have to add some channel only and AfterOriginalAirdateOnly modifications as well... also noticed if padding is "0" it will be blank as I believe the key doesn't come through in json, so you may wish to put a 0 there), Upcoming , and the "new" tab (not mentioned here) but was bummed recordings wasn't coming through as "engines" weren't discovered (and yes there is an unsupported way to have multiple engines if the first storage location is null). The way I chose to edit was to make the modifications then click a button to submit (modify), I believe VIEW (the RUI) updates these as changed... for example if I change always to recent, change start padding and change end padding I submit once where it appears it makes 3 requests in VIEW (I keep track of changes made to only submit the parts changed as I cheated on my AOAO to subtract a day from the date rather than figure out what midnight UT the day before was and was sometimes having problems with the date functions using UT like they were supposed to).

I'll probably work on cleaning up my javascript implementation to make it more readable and share with a couple of people (I wrote it as a single file complete with script and css and it would be much better broken up into parts).

I am actually really enjoying the simplicity and clarity of the APIs they've exposed, they're a great foundation to fix all the shortcomings as the official client / add-on.

That is exactly what I felt and renewed my personal interest in programming for "fun" (I haven't really done any I consider decent in nearly 25 years that was just for fun except maybe something concerning hardware like with a defcon badge... I learned on punch cards and even writing code in octal/hex so at a time that was actually easier for me).

[avdp]

was bummed recordings wasn't coming through as "engines" weren't discovered (and yes there is an unsupported way to have multiple engines if the first storage location is null).

The engine discovery is done though my.hdhomerun.com. However there is a "if" statement in dvrui_hdhrjson.php to only use a discovered engine if the IP of the engine matches the web server's IP address. Easy to comment out if you want to use the php on a stand-alone server. The idea there is that this package is intended to be used to manage a recording engine instance on the same box. In hindsight this is probably a bit short sighted, it was also the quick way out of having to deal with potentially multiple engines getting discovered.

[demonrik]

was bummed recordings wasn't coming through as "engines" weren't discovered (and yes there is an unsupported way to have multiple engines if the first storage location is null).

Your engine might have been discovered fine. There is a "if" statement in ui/include/dvrui_recordings.php to only use a discovered engine if the IP of the engine matches the web server's IP address. Easy to comment off if you want to use the php on another server. The idea there is that this package is intended to be used to manage a recording engine instance on the same box. In hindsight this is probably a bit short sighted, it was also the quick way out of having to deal with potentially multiple engines getting discovered.

Not shortsighted, but purposeful.
The aim here is to manage the DVR engine on a QNAP Nas. So dealing with remote engines isnt required.
And as @avdp puts it, it nicely avoids having to deal with many dicovered engines.

Lots of updates in the works, and new capabilities as well. So stay tuned

Few more edits and will do a new package

OK getting around to it a little this weekend... got latest from git as a zip and put UI onto a single bay my cloud... however I am testing on one not running the engine... what I had hope would have been for engine discovery (the new way from my.hdhomerun.com) and pulling in the ip:port of that to manage recordings... I'll have to take a closer look at it to see how I want to change this (I realize your main purpose at first was probably status, start, and stop the engine so it is to be expected and it is only recently that the record engine info was online and one had to set it and most don't know their port). Keep in mind that this device isn't meant to install packages on, I totally ruined php on the device one time trying to do an update for a utility I needed for a web page as it uses the wrong page size for the version it is on so everything must be recompiled.

As mentioned, small changes will allow remote detection. Will look through it later and provide hints.

One question since I haven't looked at it much to absorb it but are requests made from the browser using javascript
(at a glance this seems to be) or are they posted to the server to make the request and/or the server initially making the discovery and getting the DeviceAuths... the advantage of the later is that one can do things more remotely (particularly with latest addition (I won't mention that tab)) but it makes the server (in your case qnap) do all the gets from SD rather than the client browser... I also saw a sleep or two in the scripts (I'm debugging through browser so not looking at the php at this time, just making notes of what is working or not) and I think it would be preferred to use either callbacks or promises than the sleep.

Server makes all the requests. I agree on the sleeps.. Will be removing as we refine/tweak.

So far it seems to work well for the Rules (though I would like to edit there and will have to add some channel only and AfterOriginalAirdateOnly modifications as well... also noticed if padding is "0" it will be blank as I believe the key doesn't come through in json, so you may wish to put a 0 there),

That's the intention over time. Have gone broad first, now time to drop drive on each page.

Upcoming , and the "new" tab (not mentioned here) but was bummed recordings wasn't coming through as "engines" weren't discovered (and yes there is an unsupported way to have multiple engines if the first storage location is null). The way I chose to edit was to make the modifications then click a button to submit (modify), I believe VIEW (the RUI) updates these as changed... for example if I change always to recent, change start padding and change end padding I submit once where it appears it makes 3 requests in VIEW (I keep track of changes made to only submit the parts changed as I cheated on my AOAO to subtract a day from the date rather than figure out what midnight UT the day before was and was sometimes having problems with the date functions using UT like they were supposed to).

As we refine, we have a select box on each entry.. Intention here is that once you select a rule you can make as many edits on that or any seleted entry and there will be global buttons to push all changes at once.

I'll probably work on cleaning up my javascript implementation to make it more readable and share with a couple of people (I wrote it as a single file complete with script and css and it would be much better broken up into parts).

Cleanup and comment would be good

[avdp]

One question since I haven't looked at it much to absorb it but are requests made from the browser using javascript
(at a glance this seems to be) or are they posted to the server to make the request and/or the server initially making the discovery and getting the DeviceAuths... the advantage of the later is that one can do things more remotely (particularly with latest addition (I won't mention that tab)) but it makes the server (in your case qnap) do all the gets from SD rather than the client browser...

As @demonrik indicated, all the work of interfacing with my.hdhomerun.com and storage server is done server side. Personally, I think that's the way it should be for a couple reasons - if you want to expose this webpage outside of your firewall (ie to manage recordings away from home) it's the cleanest, most foolproof way to avoid cross-domain security restrictions and whatever remote network/firewall restrictions may exist. The second reason is that pushing heavy duty processing to the client is not necessarily the best for mobile devices.

[demonrik]

was bummed recordings wasn't coming through as "engines" weren't discovered (and yes there is an unsupported way to have multiple engines if the first storage location is null).

ok,
the matching to server IP is here in include/dvrui_hdhrjson.php

Code: Select all

			if (array_key_exists($this->hdhrkey_storageURL,$hdhr)) {
				// this is a record engine!
				
				//get the IP address of record engine.
				$hdhr_ip = $hdhr[$this->hdhrkey_localIP];
				// Split IP and port
				if (preg_match('/^(\d[\d.]+):(\d+)\b/', $hdhr_ip, $matches)) {
				    $ip = $matches[1];
				    $port = $matches[2];
				    // if IP of record engine matches the IP of this server
				    // return storageURL
				    if($ip == $myip){	
					$this->storageURL = $hdhr[$this->hdhrkey_storageURL];
					continue;
				    }
				}

You need to remove the matching..
BUT because the rest of the code ONLY looks for a single storageURL you will need to add something to decide which one to save.
Otherwise you would need to modify recordings to check each Record Engine and remove duplicates if a recording appears in BOTH engines. Yes, since you can fudge it to have 2 engines looking at the same content through the use of NULL on one of the engines, you can also potentially have either engine report the exact same recording.

[demonrik]

Removed - see OP for correct version and link

[avdp]

Please see the following post regarding the Web UI.

https://www.silicondust.com/forum/viewt ... 19&t=37869

[rcgfl]

I recently purchased a QNAP HS-251+ and am using it to replace an older MyCloud NAS that originally was running my DVR recording engine. I installed the recording engine via the q0.8.8 package. I also installed the new DVR UI through the link provided within the post linked above, using the QNAP as my web server (UI version 0.11.1). My HDHomeRun Prime is running the most current firmware (20160630atest2). All of the default locations for the config file location, record path and port are being used, and confirmed via the UI still provided with the package installer (UI version u.0.9.0). I can also verify through the UI that HDHomeRun RECORD is running (record engine version 20160630atest 2). Via either web page, I am unable to see any recordings. I am able to see recordings through Kodi, the Windows 10 app, and other iOS / TVoS apps (Channels Beta for Apple TV if I use an older record engine, InstaTV beta for Apple TV and iOS). The DVR Web UI provided via the other thread reports that 0 recording engines are found. Going through my.hdhomerun.com/discover, no StorageID or StorageURL is shown. Uninstalling the package then reinstalling it does not change this. I had earlier used ssh to rename the the installed record engine and copy an older record engine (20160426) to maintain compatibility with the Channels app, and thought that might be the issue, but neither replacing that record engine with the one initially provided with the package installer, nor reinstalling the package helped. Any help that can be provided to get this to work would be appreciated.

[avdp]

Ah, I see the story is getting a bit more complete now.

I don't know of a way to force my.hdhomerun.com/discover to refresh its information. From my own testing, I've had a server that was shutdown show up for many hours. And it's taken up to an hour for a new server to show up. I know @demonrik has had his server disappear sporadically, apparently restarting it fixed it. SD has acknowledged the issue and said a future DVR release should fix.

Long way to say... This software is still beta.