FreeBSD for RECORD platform

Want to write your own code to work with a HDHomeRun or work with the HDHomeRun DVR? We are happy to help with concepts, APIs, best practices.
averyfreeman
Posts: 107
Joined: Sun Apr 09, 2017 11:12 am
Location: Olympia, WA
Contact:

FreeBSD for RECORD platform

Postby averyfreeman » Sun May 27, 2018 1:05 pm

FreeBSD with ZFS for Recording TV using HDHomeRun tuners

Disclaimer: I can't make any promises that any of the included information is correct, just that it appears to be working for me.

This is for an HDHomeRun RECORD / DVR setup with 2x HDHomeRun Prime digital cable TV tuner boxes, which use cable cards from the cable company to tune channels (3 tuners each). The host RECORD system records and plays back MPEG-2 streams to anywhere on your LAN (or further if you want to blow holes in your external firewall)

tl;dr
Background: I had been using ZFS for a 4-drive pool in CentOS 7, and it had been working well for recording the TV, but Samba appeared to having issues and the share functionality built into ZFS doesn't work in CentOS (known bug, unlikely to be resolved).

Although it was hard to leave the 10-year lifecycle of CentOS, I built a VM to import my zpool to with FreeBSD 11.1-RELEASE on it. The ZFS is more mature on FreeBSD, so I thought it'd be a better platform for serving my SMB shares -- maybe ''zfs set sharesmb=on'' shares will even work now (update: it doesn't - it only works on Solarish systems - but Samba throughput has been exponentially better).

Additionally, the general userland Samba seems to work quite a bit better, too. On CentOS 7 my network share sequential transfer speeds were crawling at 1.2 MB/s. It was painfully slow. On my FreeBSD VM with Samba 4.5 connected to a Windows 2016 domain mounted in Windows to a Thinkpad T420 over a MoCA 2.0 double-bonded connection I was able to get over 100MB/s sequential transfer speeds.

This is using the same configuration settings, on the same network, too. It's literally 100 times faster.

The setup: I started by creating a dual-vdev RAID 1+0 configuration on four HGST 7k2000 drives, as it's a good combination of speed and redundancy. Also, mirrored vdevs have a few other advantages: their scrub and resilvering speeds are much, much faster.
They can also be expanded just by adding additional vdevs - pop in a couple more HDDs and add them to the pool and it keeps growing.

While general ZFS commands are beyond the scope of this post for now, I want to quickly mention the ZFS command for that type of array was:

# zpool create mirror ada1 ada2 mirror ada3 ada4 nameofpool

I like to mount it in the /mnt directory instead of root.

# zfs set mountpoint=/mnt/nameofpool nameofpool

This is important - read up on it before you make your pool: Thankfully, ZFS either detected the ''ashift'' and implemented the value properly (ashift=9 - these are old 512b sector drives) or ashift=9 was just default. I'm not sure. If you are concerned about this (and you should be), especially if you have 4k block-size "advanced format" drives, be sure to add ''ashift=12''. Google ''ZFS ashift'', there is a ton of information and discussion. I pretty much just got lucky.


Obligatory OS and pkg manager updates:

# freebsd-update fetch install

# pkg install


Slight linixification: There are some little things Linux users might discover about FreeBSD being similar but different unnerving since they're used to things working a certain way. A lot of this can't be avoided, but for a few things that can, here's what you can do:

# pkg install sudo bash nano

# chsh -s /usr/local/bin/bash

Then edit /home/local/.profile and /root/.profile for preferred editor (It's set up for vi like most Unix-ish OS. If you prefer to use nano or vim or emacs, this is how you set it as your user's default).

Include bash only if you prefer bash to sh. From what I understand it's backward-compatible with sh. Note: if you are running sh scripts you may wish to invoke $ sh scriptname.sh rather than $ ./scriptname.sh to make absolutely sure. Also, remember bash uses export while sh uses setenv.

Oh, and be sure to make a local user by invoking adduser'' - I made one called local so as not to conflict with my domain usernames - because as far as I can tell, there is NO way to SSH to FreeBSD as root (it's a security precaution that can't be overrided in current versions of ''ssh_config'' I've tried many times and failed miserably.

Using Plex for dlna: I had originally used minidlna for my HDHomeRun VIEW server, but I find Plex works better - less chance of stuttering, etc. I pretty much just use its dlna functionality, but you still have to set it up. It's beyond the scope of this post, but remember when you're setting it up to configure your network under the allowed list of ip addresses, otherwise you'll get an endless 'searching for plex media server' next time you log in. It's located at:

Settings -> server -. Network -> List of IP addresses and networks that are allowed without auth

For example, I added 192.168.1.0/24 so any computer on my local subnet can access the plex menu.

The installation: Pkg is nice because there are different versions of the same software in the default repo. For Samba, I recommend using one-version-back, because I tried to use the latest version and it broke from a routine upgrade. Kerberos newest version is probably fine, but since I stay a version back in Samba, I stay a version back in Kerberos, too.

As of this writing, the latest versions are samba48 and krb5-116, so I install:

# pkg install \
samba47 \
krb5-115 \
plexmediaserver \
inotify-tools \
mrouted \
pimd \

Then set up the configuration files. In my examples there are lots of # comments - if you don't want to use them, just don't add them - but they can be useful for trying things out.

The first big one is /boot/loader.conf

# -------- For uPnP ------- #
ip_mroute_load="YES"
gateway_enable="YES"
routed_enable="YES"
routed_flags="-s"
# ---- For NTFS-3G -------- #
fuse_load="YES"
# ------- ZFS flags ------- #
#zfs_load="YES" # Used for booting from ZFS pool
#vfs.zfs.prefetch_disable=1 # Use if you have 4GB mem or less
#vfs.zfs.arc_max="256M" # Use if you want to limit your ARC usage
#vfs.zfs.vdev.cache.size="64M" # Use if you want to limit your cache size

And next is /etc/rc.conf

# ------- ZFS Array ------- #
zfs_enable="YES"

# -------- Network -------- #<br>
# ---------- IPv4 --------- #<br>
hostname="fbsd111.happy.hut"
ifconfig_vmx0="inet 192.168.1.54 netmask 255.255.255.0"
#ifconfig_em0="inet 192.168.1.54 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

# ------ Kill IPv6 -------- #
ip6addrctl_enable="NO"
ip6addrctl_policy="ipv4_prefer"
ipv6_activate_all_interfaces="NO"
ipv6_enable="NO"

# ----- Enable IPv6 ------- #
#ipv6_enable="YES"
#ifconfig_vmx0_ipv6="inet6 accept_rtadv"
#ifconfig_em0_ipv6="inet6 accept_rtadv"

# ----- VMWare Devs ------- #
vmware_guest_vmblock_enable="YES"
vmware_guest_vmhgfs_enable="NO"
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"

# ------- Services -------- #
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
plexmediaserver_enable="YES"
pimd_enable="YES"
hdhomerun_record_enable="YES"
samba_server_enable="YES"
winbindd_enable="YES"
#winbindd_enable may or may not necessary for your version of samba for domain membership

Please look up any of these flags before you opt to use them, the man pages in FreeBSD are extraordinary and I'm sure there are lots of references and discussion about each one on the internets.

In my particular .conf files there are a lot of flags which are extraneous that can be commented/uncommented if wanting to troubleshoot the configuration easily. For instance, I have settings for mrouted, pimd and ipv6. I turned off ipv6 because it was making it difficult to secure domain membership. pimd and mrouted are for multicast which should improve playback speeds.

There's also a lot of flags I included that are VMWare-specific, so if you're using bare metal or a different hypervisor, obviously you can just ignore those. If you're using VMWare-any-hypervisor (ESXi, vSphere, Workstation, Fusion, etc.) and haven't already, grab open-vm-tools-nox11 through pkg.

HDHomerunRecord Stuff: The HDHomeRun RECORD software can be dowloaded here on their forum: viewtopic.php?t=30386

It also explains how to use the HDHomeRun RECORD service in the most straightforward way, which is:

Just download it to your media folder with wget or curl, make it executable, create a quick config file and then run it:

# cd /path/for/recordings

# wget http://download.silicondust.com/hdhomer ... rd_freebsd

# chmod u+x hdhomerun_record_freebsd

# echo 'RecordPath=/path/for/recordings/' > hdhomerun.conf
# sh hdhomerun_record_freebsd start

Now let's start plexmediaserver, too:
# service plexmediaserver start

You won't have to do this if you've restarted your computer after adding plexmediaserver_enable="YES" to /etc/rc.conf

If you look at your network in Windows Explorer you should see "Plex Media Server" under "Media Devices"

If you want to have a service for the HDHomeRun RECORD service, I put one together in a pinch by renaming hdhomerun_record_freebsd to hdhomerun_record, copying it to /usr/local/sbin/ and creating the following script /usr/local/etc/rc.d/hdhomerun_record:


#!/bin/sh
#
# PROVIDE: hdhomerun_record
# KEYWORD: stop
#
# Add the following line to /etc/rc.conf[.local] to enable RECORD:
#
# hdhomerun_record_enable="YES"

. /etc/rc.subr<br>

name=hdhomerun_record
rcvar=hdhomerun_record_enable
config_file=/usr/local/etc/hdhomerun.conf

load_rc_config $name

hdhomerun_record_enable=${hdhomerun_record_enable-"NO"}

command=/usr/local/sbin/hdhomerun_record
command_args="start foreground --conf=/usr/local/etc/hdhomerun.conf"

run_rc_command "$1"

This script has allowed me to run the program anywhere since it's in the path, but more importantly, has allowed me to stop and start using the basic service commands, and also start the service on boot in rc.conf using ''hdhomerun_record_enable="YES"''

Here's an example of it in use:

# service hdhomerun_record status
hdhomerun_record is running as pid 637.

# hdhomerun_record status
HDHomeRun RECORD running

Also, don't forget to copy hdhomerun.conf with the correct record path to /usr/local/etc/ and add hdhomerun_record_enable="YES" to /etc/rc.conf.

Now go to your HDHomeRun Setup program that you're using for configuring your network (In my case it's on an ASUS P8H61-ITX w/ i5-2500T running Windows 10 under my TV) and select "record to NAS", apply and save the settings. If the FreeBSD computer doesn't show up, don't worry - it should still work. I got the icon to appear using pimd, mrouted and minissdpd, but it's not necessary so long as your RECORD service is running on the recording computer - check it with

# hdhomerun_record status

Samba stuff for Windows Networking: I've added my Samba configuration for reference, but for more in-depth info, this site is very easy to follow: https://blog.iandreev.com/?p=2676

Note: The version he used is older than 4.6. There was a syntax change in smb4.conf for v4.6 and newer that was requires re-writing part of the configuration for domain members. If you are joining your FreeBSD system to an existing domain, please familiarize yourself with the definitive guide at the official Samba wiki: https://wiki.samba.org/index.php/Settin ... ain_Member

An example of the changes are things like winbind nss info = rfc2307 instead of unix_nss_info = yes

Here's my samba config - comments in samba are written with ; rather than # - /usr/local/etc/smb4.conf:

[global]
netbios name = FEBMARCH
security = ADS
workgroup = WEBTOOL
realm = WEBTOOL.SPACE
client use spnego = yes
client signing = yes
server signing = yes
client ntlmv2 auth = yes
encrypt passwords = yes

log file = /var/log/samba4/%m.log
log level = 1

idmap config * : backend = tdb
idmap config * : range = 3000-7999

idmap config WEBTOOL : backend = ad
idmap config WEBTOOL : schema_mode = rfc2307
idmap config WEBTOOL : range = 10000-999999
idmap config WEBTOOL : unix_nss_info = yes
idmap config WEBTOOL : unix_primary_group = yes

;winbind use default domain = yes
winbind enum users = yes
winbind enum groups = yes
winbind offline logon = yes
winbind nested groups = yes
winbind refresh tickets = yes
winbind nss info = rfc2307

;winbind nss info = template
;template shell = /bin/bash
;template homedir = /home/%D/%U

[videos]
path = /qpool/videos
write list = @WEBTOOL\"domain admins"
read list = @WEBTOOL"domain users"
writable = yes
browsable = yes
read only = no
guest ok = no
public = no
create mask = 0775
directory mask = 0775

[hdhomerun]
path = /qpool/hdhomerun
write list = @WEBTOOL\"domain admins"
read list = @WEBTOOL\"domain users"
writable = yes
browsable = yes
read only = no
guest ok = no
public = no
create mask = 0775
directory mask = 0775

You can omit that whole part under realm = to the blank space - I included that for added security, it's the first thing I would scrutinize if you are having any trouble with connecting as a domain user. Well, maybe after checking /etc/krb5.conf and /etc/ntp.conf, restarting ntpd and checking to make sure the times are all synchronized on your domain controllers, members, hosts, and hardware clock (very important).

The unix permissions underneath these shares are ''0777'', which anyone can change, even guests -- IF they're logged into the server as a Unix user. BUT, domain guests are prevented from accessing the share from the network due to ''guest browsing = no''. Domain users are allowed to view the share and domain admins are allowed to change it.

I'm sure there's a better way to get Windows ACLs and permissions settings working, but nobody's going to be logging in directly to my FreeBSD VM, so this does the trick for now without being too complicated. This seems like a good method to follow but I haven't had the patience to sit with it yet: https://daniel.washburn.at/howtos/freeb ... zfs-recipe

After a couple months of using it, seems to work extremely well. Thanks to all the help from people on this forum and the great developers at SiliconDust!

cqaigy
Posts: 18
Joined: Wed Mar 02, 2016 9:48 pm

Re: FreeBSD for RECORD platform

Postby cqaigy » Sun May 27, 2018 4:01 pm

Nice write up. Was looking to see if there was a like button and almost reported your post. :lol:

ebo
Posts: 468
Joined: Thu Jan 07, 2010 6:58 pm

Re: FreeBSD for RECORD platform

Postby ebo » Mon May 28, 2018 10:56 am

Make sure you keep a copy to repost as needed, because unless the forum managers give it a sticky (which they should) it will scroll off in 90 days.

averyfreeman
Posts: 107
Joined: Sun Apr 09, 2017 11:12 am
Location: Olympia, WA
Contact:

Re: FreeBSD for RECORD platform

Postby averyfreeman » Mon May 28, 2018 4:20 pm

Cool, thanks for the tip - I actually did edit it off-site before posting it, got lots of captchas probably for looking like spam ;)

I am going to split it into pieces and submit it to the FreeBSD forum, too, for critique. Hopefully I can tighten it up a bit more and get some more people's ideas in there.

gtb
Expert
Posts: 3475
Joined: Thu Oct 06, 2011 1:00 pm
Location: Sunnyvale, CA USA

Re: FreeBSD for RECORD platform

Postby gtb » Mon May 28, 2018 4:44 pm

Make sure you keep a copy to repost as needed, because unless the forum managers give it a sticky (which they should) it will scroll off in 90 days.
+1, although if you post in the development support area I believe it lasts a bit longer. I seem to recall I posted my packaging for Linux EL/Fedora RPM systems two times before it achieved "stickydom" (no, that is not a word).

I still believe that SD, when they get a bit further along in the release support timeline, will be providing official formal packaged solutions for (at least) the leading unix-like solutions (and ignoring the various popularity discussions, FreeBSD is arguable a contender, in its various derivatives such as FreeNAS), along with some of the leading NAS solutions (and the work demonrik has done is likely a good starting solution for QNAP and Synology). Personally, I will not lament the day when SD obsoletes all the independent works in preference to formal corporate packaging (yes, I *am* lazy, and not having work to do is the best kind of work to *not* do).

averyfreeman
Posts: 107
Joined: Sun Apr 09, 2017 11:12 am
Location: Olympia, WA
Contact:

Re: FreeBSD for RECORD platform

Postby averyfreeman » Mon May 28, 2018 11:33 pm

Gah,

I just spent the last 6 hours expanding on the post I made earlier and dividing it into six sections.

https://forums.freebsd.org/threads/how- ... -tv.66054/

It's a lot easier to follow and has a lot of updated information. For anyone who's interested...

jasonl
Silicondust
Posts: 12476
Joined: Sun Oct 28, 2007 9:23 pm

Re: FreeBSD for RECORD platform

Postby jasonl » Tue May 29, 2018 3:26 pm

Topic split and moved into Development. There's a bit too much coverage of non-DVR things to put this in the Instructions area, but if you want to make a separate post here with just the DVR stuff I can move that over to the instructions, and provide a link back to this thread for people who are looking for more overall system setup details.

averyfreeman
Posts: 107
Joined: Sun Apr 09, 2017 11:12 am
Location: Olympia, WA
Contact:

Re: FreeBSD for RECORD platform

Postby averyfreeman » Sat Jun 09, 2018 2:55 pm

Hi Jason,

OK sounds good, I actually have a split version I can put on - I don't have time at the moment but will try to get to it tonight


Return to “Development Support”

Who is online

Users browsing this forum: No registered users and 3 guests