The organisation had entered a strategic partnership with the leading hyperscale cloud provider. Significant compute and storage were available at deeply discounted rates. The strategic vision existed. What did not exist was a path from intent to execution.
Massive storage requirements
10 petabytes is a huge storage requirement but the amount of data captured per race was also accelerating. Increasing pixel counts, more camera angles and the proliferation of super-slow-motion footage (resulting in vastly more frames captured per second) meant that the raw number of bytes-per-race was steeply increasing.
A previous initiative had digitised the film and video tape portions of the archive. All this was stored in a combination of online and nearline disk storage (these held the last couple of seasons and the most frequently-accessed data from a few more years) with the overwhelming majority relegated to 8mm magnetic-tape drives on shelves. This data had all been laboriously manually tagged over the decades, not just with the circuit and year but with timestamped entries for drivers, teams, overtakes, chequered flags, collisions and anything else of note.
You could search the legacy system for all footage over the preceding decades, but for anything more than a few years old all you would get back was a tape identifier. You could request this footage be made available by sending a request off for the tape to the media team. The tape would be retrieved, loaded into a jukebox system with a robotic arm (this is not a joke) and the data temporarily put onto the online storage for you to make whatever edit you needed.
The first proof-of-concept phase of the project was to take the magnetic tape and jukebox out of the picture. In partnership with 10dot1, the creators and maintainers of the legacy storage arrays, we created a massively parallelisable, autoscaling uploading system. On-prem compute, with code and configuration managed in the cloud, would start hundreds of parallel uploading processes all coordinated by cloud-based event-driven queues and a No-SQL database. In conjunction with the business processes to support this, we were able to tune a workflow that got data off-prem and into the cloud as fast as business-as-usual would allow.
Operational resistance to strategic mandates
With the data migration in full-swing, the business were suddenly able to see the possibilities open to them. We were able to demonstrate the reality of instant, cost-optimised access to the entire archive. We became part of a first wave of initiatives that were to spearhead the transformation. However, the other teams were running day-to-day operations under sustained high-pressure delivery schedules. Their existing workflows were functional and well-understood. Further, the overarching plan, was to implement a broad, big-bang migration to a datalake. Not only was there no clear technical motivation to migration at the ops level but, as subject-matter experts, the general flaws of the datalake architecture (with which we have all since become familiar) were exposed as soon as it was exposed to the pressure of supporting actual detailed implementations.
We were able to maintain a pragmatic, delivery-led strategy that achieved the desired architectural benefits through evolving the existing system and being good corporate citizens. We could offload enterprise-level concerns like securing and permissioning the common storage in return for accepting constraints and inputs from the fledgling cloud management enterprise team. In other words, we embraced Conway's Law early and enthusiatically to build a system that would align cleanly with the emergent org chart.
Misaligned incentives and technical cultures
The programme involved multiple organisations, each with different commercial models, delivery cultures, and definitions of success. The teams from the cloud provider were highly familiar, and justly proud and confident in the capabilities their tools could provide. However, the ultimate goal was to drive adoption and spend with them (albeit saving their client enormous amounts of hassle and money in the process). Sometimes gaps would open up where "of course it's going to work" would sound a little too much like "trust me bro". In some project phases our main job became "cloud whisperer" as we explained what was happening to varying audiences. Sometimes we had to steer the dev team work in slightly less efficient ways so that the customer buy-in remained high.
Early on an extended commercial negotiation lasting eight months threatened to stall the project entirely. Maintaining delivery continuity during that period required significant risk tolerance. We made a deliberate decision to keep building in the absence of formal sign-off in the knowledge that the value we could deliver to all stakeholders would be worth it.
A permissions model that appeared unsolvable
The easy answer would have been to replicated the existing MAM system permissions. However, when we asked what the dream permission system would look like, we got a very different answer. The ideal system would grant permissions based on whether that asset satisfied one or more search queries on the asset associated with a user's group.
This created a circular dependency. You could not determine an asset's permissions until it was indexed in the search cluster, but you could not index it in the search cluster until its permissions were resolved.
We eventually cracked the problem by producing a POC lambda function that took the asset's metadata immediately before being added to the canonical NoSQL data store. We generated the search document early, then indexed that into an Apache Lucene in-memory instance within the function itself, allowing us to query the document using the same syntax and get the same relevance score as if it was already in the cluster. We enriched the metadata with the matching groups, which allowed a simple match or no-match on the existence of a NoSQL record with the asset id as partition key and group id as the sort key. This allowed us to look at an incoming request for an asset (which was accompanied by cryptographically verifiable credentials containing the user's groups) and then try and fetch the relevant record. No record, request denied. As the queries associated with groups almost never changed, this was cost effective, O(1) fast and completely accurate.