Support running multiple record engines as a pool

Downloads & Instructions
signcarver
Expert
Posts: 11083
Joined: Wed Jan 24, 2007 1:04 am
Device ID: 10A05954 10802091 131B34B7 13231F92 1070A18E 1073ED6F 15300C36
x 39

Re: NEW - support running multiple record engines as a pool

Post by signcarver »

Personally for such, I would no longer have the share in the directory of the other... I used to do it this way without issue to archive things but it would also show "everything" twice when the other was running as an engine rather than just a share. (Note network shares in that manner are not supported file locations for the DVR, but I have had good luck when one was used as an archive location but changes made to such location may not be picked up automatically after a while)

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: NEW - support running multiple record engines as a pool

Post by nickk »

FYI - it is ok to use a local path that is shared but we don't support using a remote network share.

The key problem is monitoring for file changes which is required for operation. It will work at first but once you hit about 62 folders you hit a resource limit when using a remote share.

Nick

gtb
Expert
Posts: 4224
Joined: Thu Oct 06, 2011 1:00 pm
Location: Sunnyvale, CA USA
x 16

Re: NEW - support running multiple record engines as a pool

Post by gtb »

jjm1982 wrote: Wed May 22, 2019 7:08 pm Folder 'A' is configred to be a shared folder
Not sure how you are sharing, but neither NFS nor SMB reliably work due to their lack of complete filesystem emulation. While one can sort of get things to work some of the time, the recommendation is not to share for it will fail in weird ways eventually.

jjm1982
Posts: 433
Joined: Wed Sep 21, 2011 5:07 am

Re: NEW - support running multiple record engines as a pool

Post by jjm1982 »

jasonl wrote: Wed May 22, 2019 7:36 pm The DVR holds a write lock on the log file. If you try to do that it will probably not end well. I'm not sure why you would even want to do that in the first place.
Prior to this new enhancement, I had both servers setup as a primary/backup where they would both determine if the recording engine was running on the other and if not the other would start.

Like most users, I would like to keep all of the recordings in the same location. But it isn't necessary.

It would be nice if we could configure the recording engine where to write the log file and what name it should have. But it isn't necessary, I would just need to determine if I really need to have more than one recording engine running.

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: NEW - support running multiple record engines as a pool

Post by nickk »

Feature - the HDHomeRun support for running multiple record engines also provides support for hardware redundancy.

If a server fails or drops offline, all future recordings will continue with the remaining server(s). At most the recording-in-progress is truncated.

Nick

Online
EddieP
Posts: 440
Joined: Sat Jun 08, 2019 11:04 am
x 4

Re: Support running multiple record engines as a pool

Post by EddieP »

I am testing this on 2 NAS'S everything is working great when recording only records to 1 NAS ... but I also noticed that when watching LIVETV on the same channel on 2 devices it writes the LIVETV buffer to both NAS's and uses 2 tuners.

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: NEW - support running multiple record engines as a pool

Post by averyfreeman »

signcarver wrote: Sat May 18, 2019 12:55 pm
demonrik wrote: Sat May 18, 2019 11:15 am if i was to set RecordStreamsMax to >0 is there a way to set preferred first engine?
i.e. i want to use my synology which is on the same GbE switch first
I believe nickk stated it would generally use the one with most space... I can see that I also would like to set a preference and perhaps another preference for LiveTV.
I 3rd this --

I'd like to set it up only as a failover in the event that my primary record engine is down for whatever reason, or runs out of space completely (or nearly completely)

I don't just want the secondary record engine to start recording if the primary record engine's free space gets lower than the secondary one's free space.

Hoping there's an affinity setting in the near future (e.g. 80/20 like my domain controllers), or an explicit integer for record order (e.g. 0,1,2,3)

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: NEW - support running multiple record engines as a pool

Post by averyfreeman »

gtb wrote: Wed May 22, 2019 8:39 pm
jjm1982 wrote: Wed May 22, 2019 7:08 pm Folder 'A' is configred to be a shared folder
Not sure how you are sharing, but neither NFS nor SMB reliably work due to their lack of complete filesystem emulation. While one can sort of get things to work some of the time, the recommendation is not to share for it will fail in weird ways eventually.
I have kind of an unconventional setup in that my fileserver is an OmniOS VM (for windows domain compatibility, Samba is awful) but there's no HDHomeRun record engine for OmniOS, so I use an NFS share in fstab in a FreeBSD VM, and they share a dedicated subnet specifically for the NFS share using vmxnet3 (it's an ESXi host).

I set it all up before reading that shares weren't supported, but have been running it over a year now without issue. I record the news so I can watch on delay and skip the commercials while working from home - so I do a lot of transport ops, switching between recordings, etc.

I ran a straight FreeBSD setup w/ local storage for about 6-8 months before that, and I haven't noticed a difference.

To check on the directory limit, I just ran...:

Code: Select all

$ tree -L 1 | tail -1
129 directories, 32 files
...in my recordings directory, so it looks like I've racked up 129 directories on my 4TB (3.6TB usable) 4x2TB mirror zfs array (basically raid 10).

Honestly, I'm not sure if the zfs makes a difference for large sequential reads, but I know it caches files to memory to be fed out as quickly as possible. The VM has 20GB ram allocated to an OS that probably runs in under 512MB.

The other thing is the "network connection" between the two VMs is virtual and contained within the single host. I've done iperf3 tests on it and got around 26GBps throughput, so it's not exactly slow.

I personally wouldn't want to run a share across the network for recording HDHR, even with a 10Gbps backbone, due to latency issues, and I'm not advocating anyone else do this with their hypervisor, as obviously it's unsupported. But thankfully it's been working OK so far, and is kind of interesting.

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: Support running multiple record engines as a pool

Post by averyfreeman »

I can confirm that I have two record engines working on two separate ESXi hosts in FreeBSD 12, it appears to be showing all recordings from both locations.

Thankfully it's only recording on the one I want it to, since it has more free space. Please let me know if there's an update that will allow me to set record engine priority, affinity, or something like that, so I can make it a proper failover setup.

Thanks for the great work! :D

Edit: I'm noticing the next morning using the two-DVR setup that my UI doesn't refresh new recordings unless I hit the "recordings" button. This used to happen occasionally on my single-DVR setup (maybe once every couple days?) but now happens nearly every time I view recordings.

It looks like the UI is reverting to an earlier state. I'm not sure if it has to do more with the test UI or the two-DVR setup, but it just started this morning and setting up the 2nd record engine is the last thing I've done to my setup.

Ohhh... I'm also noticing I'm on test UI 20190726b now, too - early access on (W10 1809). Last I checked I was on 0723. I'm not sure when it updated but may have coincided with this issue.

So basically enough has changed that I'm not sure what it is at this point but I'm generating logs if that helps you guys troubleshoot. It's nothing more than a minor nuisance though.

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: NEW - support running multiple record engines as a pool

Post by averyfreeman »

nickk wrote: Sun Jun 02, 2019 9:07 am Feature - the HDHomeRun support for running multiple record engines also provides support for hardware redundancy.

If a server fails or drops offline, all future recordings will continue with the remaining server(s). At most the recording-in-progress is truncated.

Nick
I tested this today:

Recording continuity didn't work for me, in that when I took my primary record server down, the recordings that were recording at the time did not continue recording

I did not wait to see if the next scheduled recordings at the top of the hour would have started on the secondary server, but I will have to test that at some point

Interestingly, the recordings that were recording on the primary record server started recording again (towards the end) when I brought the primary record server back up

The secondary record server was OK for live TV, and the recordings on that server were present - but the live TV (which I was watching when I took the primary record server down) stopped and I had to re-start the HDHR client

Still some kinks to work out? Or am I just expecting too much?

Thanks :D

signcarver
Expert
Posts: 11083
Joined: Wed Jan 24, 2007 1:04 am
Device ID: 10A05954 10802091 131B34B7 13231F92 1070A18E 1073ED6F 15300C36
x 39

Re: Support running multiple record engines as a pool

Post by signcarver »

Your expecting too much... recordings "must" start on time as there is no late start (except at start of engine when it first comes up) and as such the recordings would be truncated (though an engine restarting may pick up in progress, or you might try poking another engine when it was down, though that might not work unless it was a new order). What you were supposed to do is wait for the next set of recordings and see that they did record.

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: Support running multiple record engines as a pool

Post by averyfreeman »

signcarver wrote: Mon Jul 29, 2019 2:33 pm Your expecting too much... recordings "must" start on time as there is no late start (except at start of engine when it first comes up) and as such the recordings would be truncated (though an engine restarting may pick up in progress, or you might try poking another engine when it was down, though that might not work unless it was a new order). What you were supposed to do is wait for the next set of recordings and see that they did record.
Was kind of on a whim, I was setting up a VMDK (migrating away from vnet NFS) and just wanted to see what would happen

oh well! Never hurts to try. Would love to see proper failover capabilities eventually. that would be super cool.

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: Support running multiple record engines as a pool

Post by averyfreeman »

I'm no longer using NFS for any recording storage. If I'm running two record engines, the list of my recordings still has to be "refreshed" by hitting the recordings button again on the top menu about 1 out of every 3 times I return to the recordings menu.

It looks like the client is only seeing one of the two about a third of the time until I hit the recordings button again. The problem goes away if I stop one of the recording services.

I'm on 20190715beta1

Edit: Tonight my client stopped showing the recordings from the primary record server at all. Had to turn off the secondary (backup) record engine in order to see my recordings again. Weird.

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: Support running multiple record engines as a pool

Post by averyfreeman »

I noticed there's a new version so I thought I'd try testing the dual recorders again

20190815beta1 ... will report back

averyfreeman
Posts: 271
Joined: Sun Apr 09, 2017 11:12 am
Device ID: 1326E235 1313788A
Location: Olympia, WA
x 1
Contact:

Re: Support running multiple record engines as a pool

Post by averyfreeman »

OK tested out 20190815Beta1 and still having an issue where at first the two seem to peacefully co-exist. I might have to click "recordings" to "refresh" one of the servers because the "secondary" record server's recordings tend to appear first, but hitting "recordings" can make the "primary" record server's recordings appear without too much trouble

Well, once the recordings on the primary record server reached a certain level, clicking the "recordings" button wouldn't make them appear anymore. This is a problem because obviously that's where most my recordings are. So I had to turn off the "secondary" record server in order to see my "primary" record server again - once it is the only record server running on my network.

To re-cap, I'm running two FreeBSD 12-Release in VMs on 2 separate ESXi 6.7U1 hosts in a domain environment (2x2012R2 DCs). DCs providing AD/IDMU local dns, dhcp. Gateway/firewall OPNSense 19.2 (root hints from DCs, not domain forwarder)

Post Reply