The Slack DevEx team reduced execution time of their end-to-end testing pipeline by reusing frontend assets when no changes where made. A seemingly simple change that's undoubtedly complex with a large monorepo being worked on by hundreds, maybe thousands, of developers simultaneously and an ecosystem of integrated tools.
Identifying changes to frontend files was done efficiently by git diff
and finding the last frontend build was done using "straightforward S3 storage concepts". The latter was never elaborated on.
With hundreds of PRs merged into this repository daily, identifying a prebuilt version that was fresh enough required robust asset management. By using straightforward S3 storage concepts, we were able to balance recency, coherent file naming, and performance to manage our assets.
This leaves me to speculate. - Is the latest build always stored with the same prefix? - Is the prefix of the last frontend build recorded in a key-value store that can later be retrieved? - Is the last frontend build information retrieved from git and the prefix pieced together on retrieval? - Or is it something more complex? Probably.
Given the quote from the blog, it leads me to believe the prefix is made up of the date, with the commit hash in front for better prefix performance. It's all just a guess for now.
Read from link