We interviewed one of our clients, Michael Tuszynski, CTO at San Francisco-based Fandor, about what it was like to implement RO’s DevOps-as-a-Service solution and what it’s like to work with us.
Q: Mike tell us a bit about your background.
A: I’ve been in web technology for 22 years. I built my first website in 1995 in Netscape and Notepad. I got into ad tech about 10 years ago. The cool thing about that work was that I focused on high-performance, massive-scale ad serving. The underlying technology was very complex – we were doing Twitter scale before Twitter. By early 2008, we were serving one billion ad impressions per day. The same infrastructure had to handle multiple massive portals at the same time.
Q: What drew you to Fandor’?
A: I’ve been at Fandor two-and-a-half years. I wanted to build products and learn how to run a full-scale streaming video platform. Everything at Fandor was homegrown, so I had the opportunity to set up a real engineering practice and processes. I hired a team and grew it to the 14 engineers and 10 operations staff I have now. I spend a lot of time thinking about how to build products and optimize day-to-day operations. Every day brings new challenges. It’s a good problem to have.
Q: Tell us about Fandor’s architecture.
A: I inherited a legacy Ruby-on-Rails application, which was fine for a small startup. Then we implemented a new business plan where we promoted shorter-form content in order to gain more site visitors and subscribers. We experienced performance failures due to scalability issues and spent a lot of time patching problems. Load testing on our infrastructure confirmed that it couldn’t handle the increased volume. We were doing as much firefighting as product design and innovation.
Q: What did you do to handle the problem?
A: We started by identifying the performance improvements and efficiencies we were looking to gain, and then decided to make a radical change from a monolithic application infrastructure to a containerized infrastructure.
Q: Did you consider hiring in-house DevOps engineers?
A: Our needs vary. Our DevOps load varies. It was too expensive of a proposition for our growing company to hire one or more full-time DevOps staff that we wouldn’t use consistently throughout the year.
Q: What was it like implementing RO’s DevOps-as-a-Service solution?
A: RO put together a plan for Fandor in February 2017. We relaunched the front side of our paywall (restricted access to online content via a paid subscription) within 2.5 months, going from no code or infrastructure to three new code bases and a brand-new architecture. There was a learning curve for everyone on the team – we all gave up the comfort of what we had been doing in the past. But RO’s DevOps team designed and monitored a solid architecture, which instilled confidence in the team and gave us the freedom to focus on other business-facing initiatives.
Q: How did your team feel about Kubernetes?
Q: So you didn’t see Kubernetes as too complex for your needs?
A: Not at all. RO managed all the complexity for us. That’s why we hired them.
Q: How did your team feel about working with RO?
A: I have engineers in San Francisco, Sacramento, and Philadelphia, so we’re accustomed to working with distributed teams. However, we had had other third-party engagements, so we were initially skeptical. From the start, RO integrated their solution into our framework and acted as an extension of our team. RO’s service and responsiveness have been phenomenal.
Q: How’s the system running?
A: We implemented a worst case scenario test. We figured we’d throw lots traffic at the system to see what would happen. The scaling worked exactly the way it was designed.
At one point, our cluster encountered problems. RO notified us that they saw inconsistencies with the underlying hardware, and within hours our production cluster was rebuilt on new hardware with no outage. It’s an example of how the infrastructure is serving us well. With our old infrastructure, it would have taken a day at minimum. Our team had an aha moment here. Our systems can run great – and stay up and running every day.
Q: What’s on the horizon for Fandor?
A: We’ve laid down a solid infrastructure, and now it’s time to start building on top of it. We recently launched a search feature, and we’re now in the process of implementing new administration features and migrating our entire video streaming service onto Kubernetes. The performance has been spectacular. In the future, we’ll be migrating our iOS, Android, Apple TV, Android TV, Roku and other apps on smart TVs. We’re also looking to scale our video recommendation engine.
Q: What has it been like to work with RO?
A: From my perspective, the team comes before everything else. Strive to be part of something bigger than yourself and surround yourself with excellent people – that’s the secret of success. Now that RO is managing and monitoring our infrastructure, our team gets to focus on doing exceptional development work – the work they actually want to be doing.
In the past, it was white-knuckle time when we did a release. Not anymore. If your only tool is a hammer, every problem is a nail. We made conscious choices based on our specific business challenges and chose the right tool – Kubernetes – and the right team partner – RO.
Fandor is a subscription-based streaming service for indie films, documentaries and international features and shorts. They are committed to the advancement and preservation of film art and culture, delivering a 50% revenue share to films’ rights holders and partnering with educators, students, filmmakers, festivals, and distributors. www.fandor.com