Remote Production with Pac 12
San Francisco’s Pac-12 Networks leads the field in remote broadcasting, with a unique Production Model, which allows it to broadcast more than 850 live events every year. Over the last four years, Pac-12 Networks has developed this model of broadcasting and is now inspiring others to do the same – in fact, its model is the inspiration to many of the industry’s big-hitters, with the likes of ESPN, Sky Sports, Time Warner Cable and Fox Sports looking at Pac-12’s infrastructure to help them become more efficient.
Meanwhile, their ambition continues to grow, both in terms of scope and technology.
We spoke to Pac-12 Networks’ leaders to talk about how necessity forced them to think about how to mix live sports in a different way, how to improve quality while reducing costs, and how that method is inspiring other broadcasters to look at their own systems.
Increasingly, broadcasters are looking at IP to provide them with greater efficiencies.
IP, and AoIP, have become industry buzzwords, and the market is already crowded with different voices jostling for attention. Possibilities abound, with the promise of efficiencies across technology, personnel and content.
Pac-12 Networks in San Francisco is already way out in front. Since 2012, Pac-12 has been taking advantage of existing IP infrastructures, establishing an IP production model that has become an industry standard. It is changing the way live sports are being produced and allows for unmatched flexibility, efficiency and control, while enhancing production value across a staggering 850 live events this year alone.
The Pac-12 Conference is made up of 12 of the leading universities in the Western United States. These universities include the University of Arizona, Arizona State University, University of California Berkeley, University of California Los Angeles, University of Colorado Boulder, University of Oregon, Oregon State University, University of Southern California, Stanford University, University of Utah, University of Washington and Washington State University.
These member-institutions compete against each other across 23 college sports, and wholly own the Pac-12 Networks. In fact, Pac-12 Networks is the only broadcaster in the U.S. to be wholly owned by its members, which allows it to be flexible, innovative and quick to adapt. This is a responsibility Pac-12 takes very seriously.
“In 2012 we were challenged to build a network capable of producing a huge amount of live remote sports productions,” says Jonathan Leess, Executive Vice President, Operations. “We also have an obligation to our distributors and so in our first year we committed to cover 550 live events, 750 live events in year two, and 850 live events in year three, four and beyond”.
“It was a daunting challenge, especially considering where we are based where the pool of engineers and mobile broadcast units on the West Coast are limited to support a college network of this magnitude. In addition to the professional sports televised on the west coast like the NFL, MLB, MLS and NHL, we added hundreds of live event productions of College Football, Basketball and many other Women’s and Olympic sports events such as Softball, field hockey, water polo and lacrosse. We needed to find a way to produce as many as 12 events in one day from our 12 member universities, all at the same high standards and with dedicated television facilities and crews.”
Pac-12’s Senior Vice President of Production, Leon Schweir, picks up the story: “In our first year we had 23 events to cover in a single weekend, and it took us some time to get the network capable of televising all these. At that time we were using more OBs – for some time we were using 18 percent of all the HD trucks in the US. We needed a way to relieve this pressure, as well as keep costs down and maintain a consistent output.”
This commitment meant Pac-12 Networks was forced to look at alternative production workflows. Its pioneering solution was a remote production workflow where audio and video is acquired at the sports venues and back-hauled to one of five studio control rooms in San Francisco where they are remotely mixed for broadcast.
This reduces the number of onsite staff and minimises set up times. Pac-12 Networks’ infrastructure provides coverage of smaller sporting events with, typically, up to four HD cameras, two SD cameras, 10 to 14 microphones and two commentator positions with minimal on-site truck facilities. Individual camera feeds and audio are sent back to the studios in San Francisco over IP where production and presentation is achieved.
“We quickly discovered that all of our universities were already connected by an IP backbone for academic file transfers and admin, and we could leverage that network by adding direct connections to our studios in San Francisco,” says Schweir. “Leveraging our universities’ Internet 2 platform enabled Pac-12 Networks to create the largest private IP Network in the US dedicated to sports. It consists of more than 5,700 fiber miles and has a total network bandwidth of 40 GB per second.
“Pac-12 Networks was the first broadcaster to use IP for broadcast in this way, and by year two we were mixing 125 events remotely. This year, our fourth, we covered 850 events – 500 with full mobile units (with transport over IP), and 350 using our remote IP infrastructure - in other words, backhauling the content and mixing it locally in the studio.”
The first problem Pac-12 Networks set out to solve was one of interconnectivity and transmission. With a robust IP infrastructure already in place, it needed a way to get everything onto it. Michael Harabin, Vice President, Technology, Engineering & Media Management, is responsible for assessing new and emerging business technologies for divisions at Pac-12 and is charged with developing technologies within the organisation. Following this work at Pac-12 Networks, Harabin and his team have become recognized as industry leaders in developing internet-based signal transportation for HD broadcasts, a centralized production model for live remote events.
“We first developed the Mobile In-Out (MI/O) unit for transmission. This 10u unit is sent out with the remote trucks, and we also have them stationed at all our on-campus athletic venues. This unit includes mic amps, DSP, embedding, encoding of IP streams and comms,” he says.
“We have an equivalent set of hardware in the studio in San Francisco. These provide a wireless access point for the production team in the field directly to our network, as well as four digit extension dialling for our phone system. It has super low latency, it is stable and very low cost. A key feature is that DSP in the MI/O generates IFBs for local commentators, which negates any latency issues to and from the production centre in San Francisco when creating monitor mixes. This solves the latency problem which was experienced in the early trials at Big Ten Network, while operational latency for things like camera tallies is almost non-existent.”
“Utilizing our MI/O unit and a small sprinter van, we are able to connect the Pac-12 Studios in San Francisco to 97 of our on campus athletic venues and transmit video and audio bi-directionally as data. This essentially turns our San Francisco Studio into an onsite TV truck. There are no video circuits and no satellite uplinks – all connectivity is achieved via IP.”
Back at the Studio, audio is picked up and mixed on one of five Calrec Audio consoles, but the remote production games are all on one of three Calrec Summas – again, Pac-12 Networks was ahead of the curve and were the first broadcaster to install the Summa in the United States. Their audio team worked alongside Calrec to ensure the Summa console met their exact requirements.
Pac-12 Networks got smarter still. Like many other broadcasters it maintains levels of redundancy on commercial fibre or satellite, but it also has layers across its IP platform.
“Although our IP network is very stable, we still have options over IP,” says Leess. “Our Master Control is in Denver, our studios are in San Francisco – this means we can route games either to Denver (where Master Control is located), or San Francisco (where the studios are). If we lose the network feed to the Studio we can re-route it to Denver where they integrate commercials uplink the fully produced games to our affiliates from there.”
Pac-12 Networks’ infrastructure is robust, flexible and very cost-effective, but the real benefits came later. Efficiencies in set-up time and staffing provided some unexpected bonuses, and while cost savings in this model are very apparent – average cost savings are between $12,000 and $15,000 per event - it’s actually as much about quality and consistency as it is about cost saving.
It’s as much about people as it is about technology.
“This year we produced 500 events with full mobile units in the field, with full crews and announcers, and transmission for all of these were over IP through our transmission boxes,” says Leess. “The other 350 events were using a multi-path MI/O with our smaller Sprinter vans, sending the video and audio streams to be mixed at HQ.”
“Most of the crew for these broadcasts are in San Francisco; the crew includes a Producer, Director, Technical Director, an A1, a Chyron Graphics, Bug Box (score) and a replay operator. In the field we only need camera operators, audio support, utilities and announcers. All of the equipment we have at the studio supports that small team in the field, so we can produce a show with the same high standards as a large, multi-unit OB.”
“This has allowed us to improve all our broadcasts,” adds Schweir, who is responsible for overseeing all remote and studio production for Pac-12 Networks and ensuring a consistency of look and feel. “For example, it took us almost three years before all our American football games were in 5.1 surround. This is in part due to the availability of 5.1 capable trucks, but fundamentally it’s about getting enough A1s who could mix in 5.1 surround! Such specialists are difficult to locate in some of our remote locations!
“Now we are mixing water polo, rugby, rowing and other niche sports in 5.1 surround via our remote production model because we have these skills available to us at all times in San Francisco. This model has allowed us to maintain the same high standards across the board, and it is extremely efficient. We can do all the FX here in real time, and as all the audio comes into the studio on Calrec’s Hydra2 network, it’s quick and easy to mix in 5.1.
“It allows us to provide the same crew for every event across a weekend; output is very consistent and it has allowed us to significantly enhance all our broadcasts.”
“Ultimately, this system gives us flexibility and scope,” says Harabin. “We have MI/O transmission facilities installed at all 12 universities, and we also produce events at other locations on the West Coast. As soon as our truck arrives at the venue they can connect to the MI/O unit and we are powered up and on the network…”
“In the future we can go two ways with this – we can go deeper into our production schedule and cover more events, or we could go wider and develop more services and production values which can ride on this system. A good example of this would be sharing additional content between the venue and the studio for social media content. The first time we did this was a watershed moment – engagement went up instantly, we can share things off the EVS which hasn’t aired yet. It’s important to keep pushing these boundaries to find new ways to promote the student-athletes, and the schools in order to do the best job we can.”
“We set out to be innovative and we’ve set a tone internally to strive for better,” adds Leess. “The visits we get from the major players, from broadcasters with much deeper pockets, are because they are looking at the same kind of production models to help them match the quality of their larger OB productions and achieve higher levels of efficiency. We’re all looking to increase our content without sacrificing quality.”07/12/16