Server upgrades Mar 27 + Mar 28

[nickk]

We are moving to a redundant datacenter configuration for all services.

We run about 50 servers and VMs for HDHomeRun related things.

The main TV guide database system consisting of 5 servers will be migrated over to the new datacenter tonight (Mar 27). Then tomorrow night (Mar 28) we will be replacing/upgrading all servers at the old datacenter.

We expect a disruption on guide services late at night during the Mar 27 move. Recordings scheduled to record during this time will not be affected.

There is a plan to move the ATSC 3.0 cloud transcode during the Mar 28 upgrade without disruption by making use of the existing failover system.

[ronj]

It's down right now at 7:09 pm EST! No recordings are being displayed, no discover, no tasks.... is this caused by your upgrade? It's a full blown outage.

EDIT: No guide info, no images for channels on Live, etc.

Does this mean no recordings will occur for 2 days????

EDIT: It's back working at 8:00 pm EST.

[willa04]

thats what I'm experiencing plus I have no tasks available.

[signcarver]

It's down right now at 7:09 pm EST! No recordings are being displayed, no discover, no tasks.... is this caused by your upgrade? It's a full blown outage.

EDIT: No guide info, no images for channels on Live, etc.

Does this mean no recordings will occur for 2 days????

EDIT: It's back working at 8:00 pm EST.

Too much of the ui requires internet backend which is why app may not work during such outage. This sometimes even includes accessing your recordings through the app. Recordings already scheduled should record as no outage is expected to last more than an hour or so at a time so the recording engine will know what to record as it retrieves what it is to record for at least the next 4 hours and tries to retrieve updates every hour or so but the longer the outage (particularly anything more than 3 hours) increases the likelihood it might miss something.

[nickk]

Recordings already scheduled should record as no outage is expected to last more than an hour or so at a time so the recording engine will know what to record as it retrieves what it is to record for at least the next 4 hours and tries to retrieve updates every hour or so but the longer the outage (particularly anything more than 3 hours) increases the likelihood it might miss something.

We changed that years ago - it is considerable longer than 4 hours.

[signcarver]

My point in saying 4 hours is that is what one could always depend on... I have heard longer times but have not really seem such in the logs that often so I don't depend on anything longer than that but did just confirm that something 7 hours away did work but 8 hours didn't. There is also increased likelihood that other things that can happen in that time (i.e. the engine restarting for some reason) that can make it lose what it is to record later.

My point is that the record engine typically still will record for "minor" server/internet outages lasting for at least up to 3-4 hours (though most likely up to 8) so the 1 hour down time yesterday was nothing for the record engine could not to handle.

[nickk]

The database is now running from the new datacenter. This switchover is what failed yesterday. I will post the details tomorrow for anyone interested.

[banders5144]

So channel guide data should be working now correct? Not seeing it refresh on my Shield with Prime tuner

[banders5144]

Nvm, cleared cache/data, all good now