Maintaining Hard Drive Free Space

Help and support with third-party TV software.
Windows Media Center, J River Media Center, Next PVR, MythTV, InstaTV, etc.
Post Reply
youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Maintaining Hard Drive Free Space

Post by youzernaym »

Version 2.0.0 and beyond are documented here. Everything below is outdated.

I wrote a utility (python script) to monitor disk space utilization and optionally delete recordings to maintain some specified amount of free space. Posting here in case it might be useful to others.

It can be found on github: https://github.com/jmattroberts/hdhr-disk-space-monitor

Report Mode
In the default "report" mode, the monitor reports disk space utilization periodically. The default reporting interval, which can be overridden, is 10 minutes.

Code: Select all

2020-05-01 22:26:54,844 [HDVR-4US-1TB 12345678] Total: 931.06 GiB; Used: 588.58 GiB (63.2%); Free: 342.48 GiB (36.8%)
2020-05-01 22:36:54,956 [HDVR-4US-1TB 12345678] Total: 931.06 GiB; Used: 589.44 GiB (63.3%); Free: 341.61 GiB (36.7%)
Maintain Mode
In "maintain" mode, the monitor will report disk space utilization as it does in report mode, as well as maintain a minimum amount of free disk space. It does this by deleting one recording per disk space check if less than the minimum amount of free space is available. This will continue until the minimum is made available.

When the minimum free space threshold is crossed, an "off-interval" report will be written, and then a recording will be deleted.

Code: Select all

2020-05-01 23:53:49,885 [HDVR-4US-1TB 12345678] Total: 931.06 GiB; Used: 913.06 GiB (98.1%); Free: 18.0 GiB (1.9%); Minimum Free: 18.62 GiB (2.0%)
2020-05-01 23:53:50,637 [HDVR-4US-1TB 12345678] Deleting "Keeping Up Appearances" recorded on Sun Jul 28 22:30:00 2019
The disk space checks for free space maintenance are separate from those for the report. They happen in the background at an interval determined by the amount of free disk space as of the last maintenance check. The more disk space is available, the longer it will be until the next check - up to many hours. If there is very little free space left, the maintenance checks can be as often as every few seconds. This can be observed in verbose mode, but it can get very... verbose.

Minimum Free Space
The default amount of free space to maintain is 2%. This can be overridden with a different percentage, or with an absolute number of gigabytes (GiB).

Code: Select all

hdhr_monitor_disk_space.py --mode maintain --percent-free 5
hdhr_monitor_disk_space.py --mode maintain --gigabytes-free 25
Delete Policies
There are 3 delete policies that can be applied to select a recording to be deleted.
  • Age - (default) The oldest recording is selected
  • Category - Recordings are sorted first by category, then by age within category. The oldest recording in the least important category is selected. The categories, in order of increasing importance are:
    • News
    • Series
    • Sports
    • Movies
    • Specials
  • Priority - The recordings are sorted first by recording rule priority, then by age within priority. Recordings that have no associated recording rule are given high priority. The oldest recording with the lowest priority is selected.
Watched Recordings
The delete policies described above do not take into account whether recordings have been watched or not. To have watched recordings deleted first, before the selected delete policy comes into effect, use the --watched-first option.

A recording is considered to be watched if there are fewer than 3 minutes remaining to be watched. This can be modified using the --watched-offset option.

Listing Recordings
This option is available so that, in combination with --delete-policy and --watched-first, recordings can be listed in the order that they would be deleted. This can help determine which delete policy is preferred.

No space check or deletion happens when this option is used.

Code: Select all

hdhr_monitor_disk_space.py --list-recordings --delete-policy age
hdhr_monitor_disk_space.py --list-recordings --delete-policy category
hdhr_monitor_disk_space.py --list-recordings --delete-policy priority
hdhr_monitor_disk_space.py --list-recordings --delete-policy age --watched-first
Friendly feedback and constructive criticism are welcome.

Usage Guide

Code: Select all

usage: hdhr_monitor_disk_space.py [-h]
                                  [-d DEVICE_ID|IP|HOSTNAME|ALL [DEVICE_ID|IP|HOSTNAME|ALL ...]]
                                  [-f FILE] [-m {report,maintain}]
                                  [-i SECONDS] [-c NUMBER]
                                  [-g GIGABYTES | -p PERCENT]
                                  [-s {age,category}] [-w] [-o SECONDS] [-l]
                                  [-V] [-q | -v]

Monitor disk space utilization of one or more HDHomeRun SCRIBE, SERVIO, and/or
RECORD devices. Optionally delete recordings to stay above a specified free
space minimum.

optional arguments:
  -h, --help            show this help message and exit
  -d DEVICE_ID|IP|HOSTNAME|ALL [DEVICE_ID|IP|HOSTNAME|ALL ...], --device-id DEVICE_ID|IP|HOSTNAME|ALL [DEVICE_ID|IP|HOSTNAME|ALL ...]
                        ID, IP address, or hostname of device(s) to monitor.
                        Default is "discover" which discovers devices on the
                        local network and monitors the first device found with
                        a StorageID. If "ALL" is specified, then all devices
                        found with StorageID will be monitored.
  -f FILE, --conf-file FILE
                        Path to configuration file. The configuration file
                        supports overriding the built-in defaults, as well as
                        per-device settings. See example. Per-device settings
                        are applied when a device ID is specified using
                        -d/--device-id. Options given on the command-line
                        override those in the configuration file.
  -m {report,maintain}, --mode {report,maintain}
                        Mode of operation. "report" mode reports disk space
                        utilization periodically. "maintain" mode reports disk
                        space utilization, and also maintains a minimum amount
                        of free space by deleting recordings when less than
                        the minimum amount of free space is available. Deleted
                        recordings are set to record again. Default is
                        "report".
  -i SECONDS, --interval SECONDS
                        Number of seconds between space utilization reports.
                        Default is 600.
  -c NUMBER, --count NUMBER
                        Number of space utilization reports to print before
                        stopping. Default is to continue forever. To disable
                        regular reports in maintain mode, set this to zero
                        (0).
  -g GIGABYTES, --gigabytes-free GIGABYTES
                        Minimum number of free gigabytes (GB) of disk space to
                        maintain. Only applicable in maintain mode. Cannot be
                        used in combination with -p/--percent-free.
  -p PERCENT, --percent-free PERCENT
                        Minimum percentage of free disk space to maintain.
                        Only applicable in maintain mode. Cannot be used in
                        combination with -g/--gigabytes-free. Default is 2.0,
                        if neither gigabytes or percent are specified.
  -s {age,category}, --delete-policy {age,category}
                        Delete policy / sort method. Determines how recordings
                        are sorted when selecting one to delete in maintain
                        mode. "age" sorts only on the age of the recordings.
                        "category" sorts first by category ['news', 'series',
                        'sport', 'movie', 'special'], then by age. Use in
                        combination with -l/--list-recordings to determine
                        which policy works best for your situation. Default is
                        "age".
  -w, --watched-first   Delete watched recordings first, before applying the
                        selected delete policy. Default is to apply the
                        selected delete policy without regard to whether
                        recordings are watched or not.
  -o SECONDS, --watched-offset SECONDS
                        Threshold for considering a recording "watched". This
                        is the number of seconds remaining to be watched at
                        the end of a recording below which it is considered
                        "watched". Default is 180 seconds (3 minutes).
  -l, --list-recordings
                        List recordings in the order that they would be
                        deleted in maintain mode, and then exit. Use in
                        combination with -s/--delete-policy and -w/--watched-
                        first to determine which policy works best for your
                        situation.
  -V, --version         Show version number and exit.
  -q, --quiet           Suppress all messages except errors.
  -v, --verbose         Print more informational messages. Free space and
                        delete messages are printed by default.

The interval for free space checks in maintain mode is independent from the
interval for disk utilization reports (-i/--interval). The maintenance runs in
the background at an interval based on the amount of free space found during
the last check. If there is a lot of space available, it will be a long time -
maybe many hours - until the next check. If there is little free space
available, it might be only a few seconds until the next check. This can be
observed with verbose output enabled (-v/--verbose).
[edited for clarity]
Last edited by youzernaym on Thu Jun 25, 2020 7:08 pm, edited 7 times in total.

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Re: Maintaining Hard Drive Free Space

Post by youzernaym »

A few changes have been made since the initial post a few days ago. Most importantly one of the URLs being called was using HTTP (insecure) and is now using HTTPS (secure).

So if you downloaded the script before the time of this post, please go and grab the latest version.

Thanks!

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Re: Maintaining Hard Drive Free Space

Post by youzernaym »

Added an option to delete watched recordings first. Description updated above.

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Re: Maintaining Hard Drive Free Space

Post by youzernaym »

Added the option to store settings in a configuration file. An example is available on GitHub alongside the script.

Also multi-threaded the reporting and maintenance/deleting. So now the report always happens at a regular interval, and the maintenance runs in the background. It will let you know when a recording is being deleted. You can see more of what's going on in verbose mode, if interested.

NedS
Silicondust
Posts: 1267
Joined: Mon Dec 10, 2018 12:38 pm

Re: Maintaining Hard Drive Free Space

Post by NedS »

This is pretty cool, thank you for sharing it!

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Re: Maintaining Hard Drive Free Space

Post by youzernaym »

I've made what are probably the final changes for a while, so it should be stable now. There are lots of internal changes, but only slight changes to the output, and no real changes to the behavior.

One thing to note if you are using this is that the short option for the configuration file has changed from "-c" to "-f".

Description updated above.

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Version 1.3.0

Post by youzernaym »

[1.3.0] - 2020-06-08

Added
  • Specify IP address or hostname instead of device ID
  • Support for RECORD devices/installations
  • Command-line option (-V|--version) to display version number
Changed
  • Disk space now displayed in powers of 10 instead of powers of 2 to match what is shown by the devices themselves
  • Removed some verbose device discovery messages
Fixed
  • Compatibility with recent firmware release

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Version 1.5.0

Post by youzernaym »

[1.5.0] - 2020-06-10
Added
  • Multiple devices can be monitored by a single process. Provide multiple device IDs, IP addresses, or hostnames to monitor specific devices. Use the keyword "ALL" as a device ID to monitor all devices found with a StorageID.
Changed
  • In maintain mode, if the requested count of reports (-c|--count) has been displayed, maintenance will continue to run. This allows a "quiet" maintain mode if the count is set to zero.
[1.4.0] - 2020-06-09
Removed
  • "priority" has been removed as a delete policy option. This allows RECORD devices to work in maintain mode.

youzernaym
Posts: 14
Joined: Sat Jun 08, 2019 9:18 pm

Re: Maintaining Hard Drive Free Space

Post by youzernaym »

Version 2.0.0 and beyond are documented here.

Post Reply