XMLTV guide data

Want to write your own code to work with a HDHomeRun or work with the HDHomeRun DVR? We are happy to help with concepts, APIs, best practices.
Post Reply
nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

XMLTV guide data

Post by nickk »

We are testing a new service providing 14-day XMLTV format guide data to customers who are paying for the HDHomeRun guide.

API: https://api.hdhomerun.com/api/xmltv?DeviceAuth=xxx

DeviceAuth comes from the HDHomeRun unit - for coding you get it from discover but for testing the easiest is to get it from
http://<ip of hdhomerun>/discover.json
If you have more than one HDHomeRun unit pull the DeviceAuth from each and concatenate them together.

The API will return the set of all channels across HDHomeRun units. For example if you have a HDHomeRun unit for cable and a HDHomeRun unit for antenna the API will return 14-day guide for all cable and antenna channels combined.

The API works in all supported countries.

Nick

gtb
Expert
Posts: 4224
Joined: Thu Oct 06, 2011 1:00 pm
Location: Sunnyvale, CA USA
x 16

Re: XMLTV guide data

Post by gtb »

nickk wrote: Thu Nov 28, 2019 10:13 am We are testing a new service ...
I presume you will not object, if, in my copious free time (which has about a dozen other higher priority items to accomplish just over the next three days)(*), if I choose to look at creating a full formal grabber (full wrapper) for the XMLTV project ( http://xmltv.org code at https://github.com/XMLTV/xmltv ) and submit it to that project for evaluation and possible inclusion into that project?

At first blush, it would just involve getting the current deviceid from local devices configured for that grabber (all, or a subset), and calling your service (and then dealing with the offset / days requirement to subset the data to maintain their baseline config requirements). It would seem to be fairly straightforward (but all projects look easy until the code hits the interpreter).

I presume there is no advantage to cache the data locally, as your service just returns everything available at each request.

Is there any rate limiting (too many requests per minute, too many requests per day)?


(*) Which likely means that I they only way I am going to get to it quickly is if I am trying to avoid working on something else I am should be working on.

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: XMLTV guide data

Post by nickk »

It should be simpler than that. You invoke the discover API to get the DeviceAuths, concatenate into one DeviceAuth string, then hit the URL. No need for any configuration or device selection.

Worst case the API returns more data than you need (ie data for a device you don't care about), but that is harmless and not worth making things configurable.

Nick

gtb
Expert
Posts: 4224
Joined: Thu Oct 06, 2011 1:00 pm
Location: Sunnyvale, CA USA
x 16

Re: XMLTV guide data

Post by gtb »

nickk wrote: Thu Nov 28, 2019 10:13 am We are testing ...
I note that this XMLTV output appears to fall a bit victim to the challenge of channels vs stations(*), and certain applications making assumptions regarding the differences.

In particular, at least one well known application (sort of) requires there to be two xml channel definitions, with the same (xml) attribute if "id", for channels that have different broadcast numbers because during automated setup it makes some assumptions. So, for example, if the same stations is broadcast on both tuner channel 2, and tuner channel 1002, one needs to create two <channel>'s with the different display names (as the app decodes the display names with the presumption that the third display-name is the tuner number for selection). Should that app do things different? Perhaps (I would say yes, but I did not get a vote). There is a proposal (get-lineup, implemented by just one (the more advanced) grabber at this point) that addresses the overloading of the display-name for tuner information to allow applications to better deal with that difference. It is also true, that for that well known app, manual adjustments to the local EPG configuration these things really don't matter as much.

Example (of what your service returns, multiple tuner numbers):

Code: Select all

        <channel id="US16604.hdhomerun.com">
		<display-name>JTV</display-name>
		<display-name>1037 JTV</display-name>
		<display-name>1037</display-name>
		<display-name>1067 JTV</display-name>
		<display-name>1067</display-name>
		<icon src="https://img.hdhomerun.com/channels/US16604.png" width="360" height="270"/>
	</channel>
Example (of what you service may need to return, one set of tuner numbers per group):

Code: Select all

	<channel id="US16604.hdhomerun.com">
		<display-name>JTV</display-name>
		<display-name>1037 JTV</display-name>
		<display-name>1037</display-name>
		<icon src="https://img.hdhomerun.com/channels/US16604.png" width="360" height="270"/>
	</channel>
	<channel id="US16604.hdhomerun.com">
		<display-name>JTV</display-name>
		<display-name>1067 JTV</display-name>
		<display-name>1067</display-name>
		<icon src="https://img.hdhomerun.com/channels/US16604.png" width="360" height="270"/>
        </channel>


(*) It is complicated (and arguably the XMLTV DTD contributes to the confusion). The way I try to explain it, a "station" is a programming entity which has a schedule of programs. A "channel" is a technical means of delivering a particular "station". Typically, in the real world (especially for cable), many "channels" deliver the same "station". However, in XMLTV, the <channel> xml (sort of) represents a "station".

gtb
Expert
Posts: 4224
Joined: Thu Oct 06, 2011 1:00 pm
Location: Sunnyvale, CA USA
x 16

Re: XMLTV guide data

Post by gtb »

nickk wrote: Thu Nov 28, 2019 7:09 pm It should be simpler than that. You invoke the discover API to get the DeviceAuths,
Not exactly, as you have to allow one to have a mix of tuner devices (say, OTA *and* cable), and different applications (or same application with selected tuner access) that may want one or the other (or all), and at least one well known app may "auto-magically" see unknown channels as newly available and try to add/use them, but may not be configured to access those "other" tuner numbers via that tuner.

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: XMLTV guide data

Post by nickk »

Channels vs stations - that make things much easier (it was a PITA to group multiple device-channels into one channel tag).

Updated - if you hit the URL again it should output different channel elements for each virtual channel number.

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: XMLTV guide data

Post by nickk »

gtb wrote: Thu Nov 28, 2019 8:10 pm Not exactly, as you have to allow one to have a mix of tuner devices (say, OTA *and* cable), and different applications (or same application with selected tuner access) that may want one or the other (or all), and at least one well known app may "auto-magically" see unknown channels as newly available and try to add/use them, but may not be configured to access those "other" tuner numbers via that tuner.
Traditionally you get guide based on a lineup which will usually include many more channels that you actually receive. I would have thought apps already deal with getting guide for channels that are not received.

garyan2
Posts: 64
Joined: Sun Aug 19, 2018 9:36 am
x 4
Contact:

Re: XMLTV guide data

Post by garyan2 »

gtb wrote: Thu Nov 28, 2019 8:00 pm Example (of what you service may need to return, one set of tuner numbers per group):

Code: Select all

	<channel id="US16604.hdhomerun.com">
		<display-name>JTV</display-name>
		<display-name>1037 JTV</display-name>
		<display-name>1037</display-name>
		<icon src="https://img.hdhomerun.com/channels/US16604.png" width="360" height="270"/>
	</channel>
	<channel id="US16604.hdhomerun.com">
		<display-name>JTV</display-name>
		<display-name>1067 JTV</display-name>
		<display-name>1067</display-name>
		<icon src="https://img.hdhomerun.com/channels/US16604.png" width="360" height="270"/>
        </channel>
(*) It is complicated (and arguably the XMLTV DTD contributes to the confusion). The way I try to explain it, a "station" is a programming entity which has a schedule of programs. A "channel" is a technical means of delivering a particular "station". Typically, in the real world (especially for cable), many "channels" deliver the same "station". However, in XMLTV, the <channel> xml (sort of) represents a "station".
Interesting approach that I hadn't thought of... nice. I'm one to champion putting all the channel numbers under one station (channel) just for the sake of reducing the size of the XMLTV file. This method, however, satisfies everyone I would think. The XMLTV is lighter and the amount of logic required to assemble and disassembly/identify is simpler.

EDIT: I must add... Sorry @nickk. I know I had requested combining channel numbers under a single "channel", but this method is better.

garyan2
Posts: 64
Joined: Sun Aug 19, 2018 9:36 am
x 4
Contact:

Re: XMLTV guide data

Post by garyan2 »

@nickk

I only have 2 remaining requests.

1) can you include the entityType and subType values from Gracenote in the categories. This way I can properly identify sporting events.
2) related to above is a Live indicator. Though not part of the DTD, it has become somewhat accepted to include <live /> the same as <new /> is used

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: XMLTV guide data

Post by nickk »

Live tag added (give it another hour or so for it to show up).

Will look into the types one next.

Nick

djp952
Posts: 1619
Joined: Wed Oct 01, 2008 8:46 pm
Device ID: 131EB7F7;131ED0E0
Location: Elkridge, MD USA
x 24

Re: XMLTV guide data

Post by djp952 »

I've had an initial run through this and am in the process of getting it hooked up to replace the JSON request, but I do have a few issues:

1. The HDHomeRun series ID for a program isn't exposed other than as part of the programme/icon element. Is this going to be 100% reliable? It seems hokey to extract the series ID from a URL, but I've done stranger things.

2. Can there be some type of attribute or flag to indicate which channel/display-name element is the one that will match the lineup JSON provided by the tuner(s)? Again I can likely just find the first one that's all numeric and go with that, but will this be reliable? Can I assume ATSC channels will have the channel.subchannel numeric format in one of the display-name elements?

3. The category elements seem to be much more extensive that what was previously exposed via the JSON Filter elements. Is there any type of master list that we can refer to in order to see what all the top-level categories may be set to? The reason here is that I have to map these into something the application supports. I would only use the first category for this mapping.

Are there any plans on making a JSON-based bulk data set like this available instead of XMLTV? XML is much more difficult to work with and adhering to the XMLTV DTD is seemingly preventing some important in-ecosystem reference data points from being included.

edit: How about adding another <episode-num> tag indicating the backend series ID? Perhaps like:

<episode-num system="dd_seriesid">C14938811ENWEAR</episode-num> ?

(I don't know where the dd_ system name prefix comes from, if it's not relevant how about hdhomerun_seriesid)

Thanks!

garyan2
Posts: 64
Joined: Sun Aug 19, 2018 9:36 am
x 4
Contact:

Re: XMLTV guide data

Post by garyan2 »

Just to add this here for those looking at the XMLTV file and seeing the dd_progid for the first time. The pieces of the dd_progid mean something. The prefixes are “MV” for movies, “SH” for shows, “EP” for episodes, and “SP” for sports. I have never seen dd_progid defined for the XMLTV, but it has always been used with the TMS (Gracenote) IDs with a ‘.’ separator.

Episode
“EP<series id>.<production number>” where <series id> is 8 digits and the <production number> is 4 digits.

Series/Show
“SH<series id>.0000” where <series id> is 8 digits and the production number is all zeros. Typically the SH data will be the generic series information. It could just be a placeholder until the episode specific information gets entered in the listings, or it could be something like a daily news broadcast which will not continually update the descriptions or production numbers.

Movies
“MV<movie id>.0000” where <movie id> is 8 digits. From what I have seen, the same movie can have different IDs for the different languages, rather than using the same ID and just adding the titles/descriptions/metadata for the different languages to it.

Sports
Sports is tricky in general. They can be “SP”, “SH”, or “EP” which makes things fun. You will have to rely on other data provided for the program to determine what it is. If we can get the entityType and/or subType, that will make it easy to identify sports events and sports shows (non-events).

djp952
Posts: 1619
Joined: Wed Oct 01, 2008 8:46 pm
Device ID: 131EB7F7;131ED0E0
Location: Elkridge, MD USA
x 24

Re: XMLTV guide data

Post by djp952 »

@garyan2, do you happen to know if there is a reference point available for what I assume are the Gracenote genres coming in via the Category tags? Just scanning my own data I have 97 unique top-level categories, which is quite a lot to deal with, and I'm sure it's a small subset of what's possible. The old JSON data had far fewer to deal with in the "Filter" element.

The granularity is GREAT, don't get me wrong, but I need to try and map these categories into DVB-SI content descriptor flags (refer to Table 17 of https://www.etsi.org/deliver/etsi_en/30 ... 10301p.pdf) for the Kodi application's EPG grid as best I can.

Your post above about using the dd_progid to get a basic idea of what to set it to was very helpful, thank you, I never thought to use that before! Using this gives me at least a basic DVB-SI mapping for the program until I can figure out what to do with the Category values.

If not, no worries, you just seem rather knowledgeable about where the source data originates :) Thanks!!

garyan2
Posts: 64
Joined: Sun Aug 19, 2018 9:36 am
x 4
Contact:

Re: XMLTV guide data

Post by garyan2 »

I do have an old document from TMS laying out the schema for their XTVD documents. It has a slew of genres/categories in it for reference. I'm sure over the years some new ones have been added, but it will probably give you a 90% answer.

If you're not aware, TMS was bought/merged/whatever with Gracenote and not much was changed.

Just PM me an email address to send the PDF. I'll get it to you tonight.

nickk
Silicondust
Posts: 20163
Joined: Tue Jan 13, 2004 9:39 am
x 376

Re: XMLTV guide data

Post by nickk »

Added series-id tag (non-standard) and star-rating tag (standard).

Nick

Post Reply