Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Silicondust Blog
nickk
Silicondust
Posts: 15417
Joined: Tue Jan 13, 2004 9:39 am

Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by nickk » Fri Sep 12, 2014 11:55 pm

It has been a crazy few weeks, with an interesting dilemma.

We started seeing problems with MTR700 tuning adapters recently, but only with new installs.
No error message, just wouldn't complete the authentication process.

I have a MTR700 in my lab and it worked perfectly. Still, we had enough debug information from helpful customers to suggest there was a problem and replacing hardware wasn't helping.

We booked time with CableLabs to use their lab facilities - this gives us access to see both sides of complex cable plants and activate new installs at will. They flew their lead lab tech out to California (great guy BTW) and we hammered on everything we could think of, including testing all sorts of possible provisioning mistakes. Everything we tried worked perfectly. Some days are like that.

The weird thing was it was only new customer installs seeing problems. Going over the trouble tickets again we started noticing a pattern... every device was new enough that they all had security certificates from the same batch supplied by the signing authority. These certificates are similar to SSL.

Long shot - program a HDHomeRun dev unit with a certificate from this batch.
Motorola CableCARD - authentication passed.
Motorola MTR700 tuning adapter - authentication didn't complete.
Cisco CableCARD - authentication passed.
Cisco tuning adapter - authentication passed.

Bingo. The only change was the security certificate.

Except the security certificate was perfect.

We pulled a new unit out of stock and couriered it to Arris/Motorola cross-country overnight.
Long story short, problem confirmed to be a bug in the MTR700 firmware. It affects all security certificates issued after a certain date regardless of the vendor/product.

How things work - bug will be fixed by Arris/Motorola (probably already has), go through their QA and release process, be released to MSOs, go through their QA and release process, be pushed to devices in select markets, roll out further, and at some point everything will work again and everyone will be happy.

Question - can we do better? it may not be our bug but can we work around the problem so customers don't need to wait?

We think so, but at great cost.

When you order a SSL cert from a website it is cheap because it is fully automated. If you want to reissue SSL certificates with something non-standard about them to work around a bug nobody was aware of before it requires special handling. Think custom script development, custom signing ceremonies, etc.

Option 1: we do nothing and wait for the MTR700 firmware to roll out.

Option 2: we do the workaround. Calculation: external costs + internal costs divided by the number of customers we expect to hit the problem over the next 3 months = approx $400 per affected customer to do the workaround.

Some days you just want pick up a large stuffed-crust pizza, a tub of ice cream, and go home.

The tough decision has been made - we are doing the workaround. Signing ceremony will happen on Tuesday. Firmware update is in QA testing now. There is a pretty good chance we can release the update mid-week. If everything works as planned the MTR700 should be happy and Switched Digital Video will work for everyone as it should.

Online
signcarver
Expert
Posts: 8045
Joined: Wed Jan 24, 2007 1:04 am
Device ID: 131B34B7 13231F92 1070A18E 1073ED6F

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by signcarver » Sat Sep 13, 2014 1:02 am

I think this is great news and though those affected probably won't think so, I am surprised/grateful for the quick response as things like these can take several months instead of the 2 since it was first reported and just the one month since it seemed to become a larger problem.

It seems it will be easier for the customer to get fixed than what I envisioned which was return of the device to you to manually load a specially crafted certificates (or exchanged) and though it would leave a bitter taste, I almost thought there might be a slight fee, or at least one way of shipping, to do so as I could see this bug that wasn't SD's fault costing a lot to fix.

Is it possible to to get a device ID range of the affected units such as all those after 131C0000?

It had been mentioned that Arris has a later firmware than the 1.37 being reported already out (1.39 from Feb 2014 usually only delivered in devices after FCC complaints in new Arris branded TA's when the provider only offered 1.34) so it would be nice to know if that version fixed the problem or if it still needs to be a later version. I would like to know the version that does fix this even though you have a workaround since I support other devices as well... I do hope the FCC will step in (I never thought I would ever say that) if cable companies don't offer the new firmware in a timely manner as it may also set precedence for the cisco markets to move to 1901 as my provider still doesn't have it after it has been out for nearly a year... they have stated they have no intention of deploying 1901 and are waiting for the next version.

scottfmcc
Posts: 9
Joined: Thu Sep 04, 2014 7:22 pm

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by scottfmcc » Sat Sep 13, 2014 5:58 am

I will say thank you. Too few companies will go to the lengths required to take care of their customers. This has been handled with impressive transparency and courtesy.

mchias1
Posts: 51
Joined: Mon Nov 13, 2006 10:42 am

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by mchias1 » Sat Sep 13, 2014 7:35 am

I want to thank you all at silicon dust for reviewing and getting a fix to our problem. As a electric engineer I understand the development and testing process. I also would like to send a huge appreciation to y'all for doing a fix out of your pocket. Sounds like the issue is with arris and yet you are still willing to spend money to help your customers. I look forward to hopefully getting all my channels soon. :-)

huniper
Posts: 19
Joined: Fri Aug 08, 2014 7:11 pm

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by huniper » Sat Sep 13, 2014 8:55 am

I have dealt with other companies that have refused to acknowledge problems with their product, or that will try and pass off the issue as the users, or some other parties fault. I am very happy that SiliconDust is not one of those companies

You guys have earned a customer for many years to come. The transparency, communication, and efforts by SD have been absolutely stellar throughout this entire issue and I want to express how thankful I am that you guys are going above and beyond your responsibilities and doing everything you can to help your customers.

I am 150% satisfied with the response to this issue and there should not be a single person that can find fault with this resolution.

vrtladept
Posts: 10
Joined: Thu Aug 07, 2014 7:13 pm

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by vrtladept » Sat Sep 13, 2014 1:38 pm

As one of the original reporters of this issue who has been affected for about 5 weeks now, let me say, Yay!

Thanks for the update! I feel it's an injustice that you have to spend $400 per customer to fix Motorola's problem, there should be a way for you to recover that from them, if there was broader justice here.

Thanks again!

sdhomerun
Posts: 4
Joined: Sat Aug 30, 2014 6:22 pm

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by sdhomerun » Sat Sep 13, 2014 1:41 pm

Holy smokes. What an amazing turn of events, and what an impressive plan! As a former **** user and now a new HD Homerun owner, I'm thrilled to have purchased from a company that makes the best choices for its customers. Believe me when I say this, I'm spreading the word! Thank you!

Mediaman
Expert
Posts: 8331
Joined: Sun Aug 09, 2009 12:11 pm
Location: Olympia, WA

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by Mediaman » Sat Sep 13, 2014 2:37 pm

Just another example of SD [Mr. Head] going above and beyond. I know you won't get much press off of this but you should. Active owners like Mr. Head do not come along everyday.

While this did not affect me I've said it before; I have never received more value from a product [products actually] in my life nor a company that responds to issues faster regardless of where it came from. This issue also shows that SD will work with everyone willing to do so to resolve an issue. Great work from everyone involved! 8-)

ego2k
Posts: 13
Joined: Thu Jul 31, 2014 12:14 am

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by ego2k » Sat Sep 13, 2014 4:59 pm

Like everyone above has said. I am very grateful and amazed that there is an organization that truly supports it products and customers. Bravo! I will continue to evangelize your amazing product and organization. You truly make the world a better place. Hopefully this will result in additional sales making SD continue its success now and in the future.

Online
signcarver
Expert
Posts: 8045
Joined: Wed Jan 24, 2007 1:04 am
Device ID: 131B34B7 13231F92 1070A18E 1073ED6F

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by signcarver » Sat Sep 13, 2014 7:02 pm

Of course watch everyone complain if it doesn't actually get released this week :evil: Which is still a possibility but I am amazed that so much can be done so fast when it comes to anything involving cablelabs. If you asked me a month ago, I would have said that it would probably take 4 more months to a year unless some type of equipment exchange could be worked out.

sbaeder
Posts: 104
Joined: Sat Jan 26, 2013 9:45 pm

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by sbaeder » Sat Sep 13, 2014 9:37 pm

AMEN!

TimeWarnerLA
Posts: 14
Joined: Fri Aug 22, 2014 9:37 pm

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by TimeWarnerLA » Sun Sep 14, 2014 3:55 pm

signcarver wrote:It had been mentioned that Arris has a later firmware than the 1.37 being reported already out (1.39 from Feb 2014 usually only delivered in devices after FCC complaints in new Arris branded TA's when the provider only offered 1.34) so it would be nice to know if that version fixed the problem or if it still needs to be a later version.
Timewarner pushed 1.39 to my device and actually called to try to see if it solved the problem. 1.39 failed to work for my device.

Online
signcarver
Expert
Posts: 8045
Joined: Wed Jan 24, 2007 1:04 am
Device ID: 131B34B7 13231F92 1070A18E 1073ED6F

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by signcarver » Sun Sep 14, 2014 4:34 pm

Thanks for letting me know of that... I thought it might work since it was basically released the same day new certificates would be be bad which made me think that might be the reason for 1.39 rather than the name change from moto to Arris. Unfortunately not much is known about it and I guess it won't work.

TeddyR
Posts: 773
Joined: Fri Jul 08, 2011 4:53 am

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by TeddyR » Tue Sep 16, 2014 2:43 pm

Although I am not affected by this bug and I too am glad that SD is implementing a workaround for those affected. The thing is that it is still a small band-aid that unfortunately may delay the REAL solution from being implemented appropriately.

This issue needs to be fixed on the MTR700 side and not the other devices like the Prime. It may affect the Prime for now, but this can potentially affect Tivos and the "Other" network tuners out there as well... It just happens that SD noticed it quicker (probably due to the volume of devices out there and that stock with the new certs is being shipped constantly not like the other cablecard :lol: tuners)...

homerjr43
Posts: 3
Joined: Tue Aug 26, 2014 11:28 am

Re: Silicondust Blog (9/12/2014) - MTR700 Tuning Adapters

Post by homerjr43 » Wed Sep 17, 2014 7:55 am

It seems that SiliconDust should have some legal recourse in order to help cover the expenses required to get the devices working. According to the FCC, the cable company must provide "compatible" equipment to allow users to use there own cable card tuners. In this situation, due to Motorola/Arris's failure, the cable companies were unable to provide "compatible" equipment. To be honest, this could be a class action lawsuit for all users affected by this problem. I personally had to pay for a cable box, in the interim, in order to receive the channels that I have paid for. Some party, whether each cable provider or Motorola/Arris is in violation of the FCC regulations. I commend SiliconDust for their efforts, but Motorola/Arris should end up paying for their mistakes. I think every affected user should be eligible to some type of compensation and I think SiliconDust should receive compensation for there efforts!

Locked