I know that posts from PDC 2008 are getting a bit long in the tooth, but I was looking over my session notes and couldn't not share it. It's a session where one of the senior MS team members that delivered the NBC Sliverlight portal for the Olympics talked about the project from beginning to end.

Read this post if you…

  • Want to see what it takes to deliver a successful project that at the highest levels of scale - from a project management perspective, the lessons learned, and the "why we succeeded" (near the bottom of the post) are golden gleanings
  • Want to see what Microsoft's experiences were in working with Silverlight – especially around innovative new streaming approaches
  • Or if you just want to pump up your technology testosterone :-) (it's seriously impressive - the stats are staggering)
  • Speakers:
    • Eric Schmidt, Director, Microsoft (good speaker by the way)
  • NBC's Olympic Silverlight Player Demo
    • adaptive streaming
    • Silverlight with Media Player fallback
    • Seek/search with almost no buffering
    • At Peak, 17 live events simultaneously
    • PIP support
    • Good content categorization and navigation
    • As seen on TV
    • Highlights factory - 2000 hours of live content total - built another 22 hours of highlights - provide additional value to what was on TV, save time
    • Published in real-time, within 10 minutes of end of event
  • Session Agenda
    • Biz imperative?
    • What was built?
    • Where do videos come from?
    • Managing the ship
    • Lessons learned
  • Business Perspective for NBC
    • NBC Perspective
      • Deliver every minute of every sport live from Beijing to web
      • Where possible, deliver HD
      • Break the mould on UE
      • Make it feel solid-state (e.g. no buffering, switch channels like on cable TV)
      • Reach broadest audience as possible
      • Leverage existing assets as much as possible to save costs
      • High dwell time
      • Quality impressions for their advertisers
      • BTW - Be innovative :-)
    • MSN involvement: Leverage portal to drive traffic
    • Stats on the system: 1.3 bilion PV, 50 MM unique visitors, 70 MM videos watched, average 27 min of viewing per sessions (vs. 3-4 min for other comparable media outlets) - HUGE differentiator - 600 MM minutes of video delivery, 5000 unique clips viewed per day by final week; 35 MM mobile views, 130K peak streams (No idea before event how much would be required so they built out 2.5 times what they actually used) - delivered 3.5 petabytes of video delivered
  • Workflow – Where do videos come from?
    • 2200 events covering 25 sports with 35 simultaneously Live feeds
    • 19 days of live production
    • 5000+ videos produced
    • Content packages
      • Live - delivered over windows media services, synched with live commentary and play by play
      • Rewind streams: VOD; adaptive streaming, creating multiple levels of BW
      • Highlights: light touch and craft edits - 50 college interns - located at 30 Rock - highlight reels -
      • Encores: broadcast replays
    • Planning complexity
      • Anticipate content/BW requirements by day & sport --> determine CPU, power, cooling, storage, ingress/egress needs >> Production schedule drove advertising schedule
      • Build was on hold until new Intel Penguin processor came out
      • Can't do full screen: IOC advert requirements, Build for 1024x768, restricted BW (40Meg DS3) out of Beijing - alexzambelli.com/blog
    • Where do Babies come from?
      • Multiple cameras at each event - routed to IBC >> HD PAL source >> NBC 40 Encoders >> 2 Win Media Services (no sweat) converting from WMD into Live HD >> pushed to New York >> DMZ >> CDNs: Limelight and Level3
      • Content management and integration - SharePoint Server - Meta data was key >> common message schema (little XML text files): video identification, scheduling….
      • Highlights factory: select out highlights, publish >> auto encode and publish out to CDNs
    • Video Player
      • 2 players - PITA!
      • JSON Schema
      • Geo Fencing >> IP based restriction > NBC only has rights for distribution in the US
      • Tokenizing so that others couldn't syndicate the content (2 min lifespan for URLs then expire)
    • 2 content players
      • Had to have hot backup >> provisioned identically
      • Dynamic traffic control with Traffic Manager
  • Managing the Ship:
    • Timeline
      • Started in June '07 - vision and scoping
      • Aug - IBC and data center design
      • 11/07 - player vision and scoping
      • 01/08 - UE design
      • 02 - traffic
      • 03 -dev full stride
      • 04 - func testing and spne up of DCN
      • 05 - beta 1
      • 06 - beta 2
    • Team:
      • DeltaTre - CMS system >> best of breed scoring for real time events from Italy
      • Schematic - did the sliverlight design and player implementation
      • Omniture for analytics
      • Keynote - WMS and adaptive streaming monitoring
      • Sliq - live stream and pub point monitoring
    • Player design
      • Wireframes became the live/updating of design
      • Moved into Blend
      • Silverlight - as needs arose, just kept dropping new libraries into the app (eg. Live streaming, or ad handling)
    • Monitoring
      • "everyone was freaking out by June"
      • NBC build "all up view" for all critical systems - every part of the core engine monitored
      • Delivery and CDN health
        • Concurrent streams
        • Outbound traffic
        • 404 errors --
        • Cache efficiency -- start in Origin, but then get pushed out to CDN's edge cache - initial user experience was poor for few minutes until cache was filled out
        • But how do you know what experience is in the last mile? >> New library in silverlight >> figured out that 51 % of people actually had more bandwidth available >>
        • Also >> real time telemetry - aggregate view of events >> drill down (SSRS) >> target core issues
      • Logging
        • Shrunk down message schema as much as possible (use attributes)
        • Doing new builds on a daily basis during the basis
      • Tool - HTTPWatch - plug in for IE - sniffs all in/out traffic on browser - similar to fiddler
  • Lessons Learned
    • RFP - scrumming builds better teams - 6 diff vendors, got everyone on the same timeline >> meet face to face (not just phone/video >> add the budget
    • Everyone should know all roles and all architecture touch points
    • Reduce complexity via common schema
    • Even when volume decreased after Olympics, unique videos viewed grew >> providing the right UI is essential to make the long tail work >> use meta data to empower navigation ("snacking") avoid typical pagination
    • Long tail delivery hides issues >> having the right telemetry to support the long tail
    • Industry needs better telemetry and monitoring (Not just Omniture for this and Gomez for that) - need to integrate the stove pipes
    • Chunked workflow presented new challenges - adaptive streaming model was whole new challenge >> solution inside silverlight to do this, but no good infrastructure to support the new workflow to create that content >> rolled this new smooth streaming into IIS 7.0
    • We left some on the table in terms of bandwidth and video profile estimates >>
    • Next Big Event was the Democratic National Convention where we bumped up and did 3Meg encoding but found the sweet spot was 1.5 meg
  • Why were we successful?
    • Windows media pedigree - strong encoding infrastructure already in place at NBC
    • Strong partners - team size of 250
    • Quality of MSN audience
    • Silverlight was ready for Primetime even at Beta 2
    • Competency, passion and resources to get it done
    • 30-40 percent take rate on downloading (typical active x is 10%)
    • About 8MM downloads
  • Questions from the Audience
    • Cache hit ratio
    • Adaptive streaming - heuristics on client which makes requests - Q1 2009
    • Dev methodologies:
      • Rifle shot test at end of may - turn everything on, live on the web
      • Scrumming
      • Test harnesses
      • CDNs were doing load testing
    • How large was the silver light app? Just under 4MB