On Democratising AI and the GPU Shortage: Part 2
White Star Capital Digital Assets Fund - Newsletter #159
On Democratising AI, the GPU Shortage and the Potential of Decentralised Training and Inference Networks
White Star Capital Digital Asset Fund - newsletter #158
By Marthe Naudts
Last week I looked at the state of the GPU industry and explained how Nvidia’s latest H100 and A100 chips are the near sole hardware behind the AI boom.
Faced with both skyrocketing demand and supply bottlenecks, even hyper scalers like Amazon Web Services (AWS) and Google Cloud Platform (GCP) cannot keep up with demand.
This leaves a multi-billion dollar opportunity for SaaS and marketplace businesses connecting disparate idle compute capacity like CoreWeave and Lambda, or more creative decentralised or blockchain-based solutions seen in the likes of Gensyn, Together.ai and Akash.
Most of these companies compete, at least initially, through lower prices than incumbent hyper scalers. This is not sustainable, and without novel value layers, these marketplaces will simply form a superfluous middle layer, all competing to secure a small piece of a fundamentally commoditised and highly divisible GPU pie.
In this piece, I will explore how companies can differentiate through technology, by clustering and coordinating disparate idle GPUs .
Clustering GPUs Across Straggling Data Centres
Managed hosting data centres are large data centre facilities that rent out rack space and bandwidth, whilst taking on the operational and financial burden of server hosting such as the cooling infrastructure and associated energy costs. Whilst cloud providers are designed to scale, legacy data centres which house the servers face fluctuating demand and therefore often are sitting on underutilised capacity.
No one has suffered more than crypto mining data centres, particularly those dedicated to mining Ethereum. Since the September 2022 Merge, Ethereum’s transition away from proof-of-work consensus left mining facilities redundant. Fortunately, unlike Bitcoin miners which typically use ASICs, Ethereum miners use general-purpose GPUs including those from Nvidia, which have a more liquid secondary market. Hive, for example, is an Ethereum miner which suffered huge losses in the immediate quarters after the Merge, and has since redirected its GPUs to support high-performance compute workloads through its HIVE Performance Cloud. Back-of-the-envelope maths suggests that at the time of the merge, with the total Ethereum hash rate at 1.03 pH/s and the average Nvidia GeForce RTX 3090 Ti hash rate at 108.75 mH/s, dividing the two leaves us with approximately 9.3m GPU units becoming available.
Marketplaces who propose to legacy data centres with idle GPUs that they could help reorchestrate servers and take over distribution to meet exploding AI demand will be met with open arms. These centres may have yet to secure the latest GPUs, but there are plenty of customers who do not need the latest and greatest. For an early-stage start-up, an academic researcher, or a public institution, speed is less important. In short, an old GPU will do the same job as an H100, it will just take a much longer time.
Clustering lower-performance GPUs to emulate the ability of a cutting-edge chip may be the best pitch to a) customers if they can secure cheaper access, b) suppliers if they can sell idle capacity which is a cost drain, and c) investors because it provides a much-needed technology layer that ensures product stickiness.
Distributed Computing Through Parallelising and Clustering Workloads
Due to RAM memory constraints, even using H100s for compute-intensive workloads like deep learning and hyperparameter tuning will require execution distribution across multiple GPUs.
Suffice it to say that clustering chips and distributing workload is an engineering challenge. A number of solutions have emerged for inter-server distribution, notably parallelising workloads through sharding model parameters across GPUs.
Some examples of this software and hardware needed for clustering include:
An interconnect solution, such as Ethernet or Infiniband, to shuttle data between the nodes
A distributed training protocol, such as PyTorch or Tensorflow. PyTorch is an open source ML framework based on the Python programming language and the Torch library. It in turn implements DistributedDataParallel (DDP) which is an algorithm that enables data parallel training. With DDP, every single GPU across every single machine will get a copy of the model and a subset of the data. The model trains through a forward and backwards pass, and then it will sync the gradients across all the GPUs. Once every process has synced all the gradients, then all the optimisers in each GPU will update the weights.
A clustering API for cluster management, such as the Message Passing Interface (MPI) or Rays APIs. Developed by Anyscale, Ray's APIs parallelise any Python code and handle all aspects of distributed execution, including orchestration, scheduling, and auto-scaling
Processes to manage the influx of data to ensure the GPUs are continuously utilised, thus enhancing their efficiency. Read our Data driven Transformation Report for more details on the future of data-mesh architecture and data lakes.
Start-ups with relationships with datacentres and crypto miners could therefore focus on making this parallelisation as easy as possible, through developing or aggregating software on the front-end, and verifying the correct hardware on the supplier side. If they can own this developer relationship, they can then expand into the entire DevOps tooling stack for distributed AI computing, which would be a very attractive moat. Our Data-driven Transformation report features in depth analysis on the innovation needed on the data infrastructure, management, and tooling needed to handle distributed AI workloads and datasets.
Next week, in the final part of this series, I’ll explore a second way in which companies can compete- building a trust layer between the two unknown entities.
🔦 White Star & Portfolio Spotlight
Exclusible co-published 2023 RECAP REPORT: DIGITAL REVOLUTION IN LUXURY & FASHION
Embark on a journey through the transformative landscapes of fashion and luxury in Exclusible’s 2023 recap report.
Alex Labs features in OKX Ventures’ 2024 Bitcoin Outlook Report
OKX highlight Alex Labs’ advantages in transaction speed and seamless bridging.
Safello launches Swish payouts
Safello, the leading cryptocurrency exchange in the Nordics, launches Swish as a payout method and by that significantly improves the process for payments when executing sell orders.
🏦 Enterprises & Institutions
Cboe exchange says Global X's spot bitcoin ETF application has been withdrawn
An application for spot bitcoin exchange-traded fund from Global X has been withdrawn, according to a filing from an exchange.
Germany Banking Giant DZ to Pilot Crypto Trading This Year
DZ Bank, Germany's second-largest bank, plans to roll out a cryptocurrency trading pilot later this year, Bloomberg reported.
Visa enables crypto withdrawals on debit cards in 145 countries
MetaMask users can now sell crypto directly to a Visa card, which eliminates the need to use centralized exchanges.
Fidelity Bitcoin ETF rakes in reported $208M, offsetting Grayscale outflows alone
Outflows from Grayscale’s Bitcoin fund slowed for the fifth day in a row, while Fidelity’s spot Bitcoin ETF saw one of its stronger inflow days since launch.
Tesla missed out on $300M profit after Bitcoin sales
Elon Musk’s auto company has liquidated 70% of its Bitcoin portfolio to date yet exhibits caution in letting go of the remaining holdings.
⚖️ Government & Regulation
Republican House leadership asks CFPB to review proposed payments rule over potential impact on crypto
The rule, dubbed the ‘Defining Larger Participants of a Market for General-Use Digital Consumer Payment Applications,’ is not clear on whether it would apply to specific digital asset entities, lawmakers wrote in a letter to CFPB Director Rohit Chopra on Tuesday.
SEC likely to approve spot Ethereum ETFs on May 23: Standard Chartered Bank
The bank predicts a potential $4,000 target for ETH if it mimics BTC’s pre-approval performance.
Republican French Hill says he's optimistic about prospect for new crypto legislation in 2024
Rep. French Hill also told reporters he is open to listen to Sens. Sherrod Brown, D-Ohio, and Sen. Elizabeth Warren, D-Mass., on their illicit finance concerns.
Harvest Fund files spot bitcoin ETF application in Hong Kong
The Hong Kong arm of a major Chinese asset manager applied for such an ETF on Jan. 26, Tencent News reported today.
European securities regulator seeks limits on non-EU crypto firms
The European Securities and Markets Authority wants to protect EU crypto firms and customers with the proposed guidelines.
💰 Funding & Exits
NAVI Protocol Raises $2M from OKX Ventures, dao5 and Hashed
To Expand the First All-in-one Lending, Borrowing, and Liquid-Staking Platform on Sui.
Ithaca announces $2.5 million pre-seed round for novel on-chain options protocol
The $2.5 million pre-seed funding round boasts a long list of prominent names and is led by Cumberland and Wintermute Ventures. Additionally, it features angel investors Andrew Keys of DARMA Capital, Stan Miroshnik of TenSquared Capital, and Georgios Vlachos of Axelar, as well as Room40 Ventures and Ghaf Capital Partners.
Bitcoin-based DEX Portal raises $34 million in seed funding from Coinbase Ventures and others
The seed round brings Portal's total funding to $42.5 million, adding to the $8.5 million raised in pre-seed funding in 2021.
Binance Labs Invests in Puffer to Support the Next Generation of Decentralized Liquid Restaking
Puffer’s first innovation received a grant from the Ethereum Foundation. Its Secure-Signer remote signing tool, allows validators to reduce the risk of slashing while enhancing capital efficiency within Puffer’s protocol.
BBO Exchange (BBOX) Closes $2.7M Pre-Seed Round Co-Led by Hashed and Arrington Capital
As the first perpetual DEX incorporating Oracle Extractable Value in its liquidation process, BBOX incorporate an innovative auction mechanism, capitalizing on latency in oracle price updates.
PayPal invests $5 million of its PYUSD stablecoin into Plaid-for-crypto startup Mesh
This investment helps to reinforce Mesh's position as a leading player in embedded finance and highlights PayPal's commitment to fostering innovation in the digital payments landscape.
Gevulot raises $6 million in seed funding for blockchain focused on zero-knowledge proofs
Gevulot plans to allocate the capital toward the rollout of its Layer 1 blockchain, which enables developers to harness ZK proofs and delegate computing tasks to an advanced network of hardware operators. This functionality creates new opportunities for scalable applications.
Forgotten Playland raises $7M to build a next generation social party game
The investment round included participation from Merit Circle, Spartan Group, C2 Ventures, and Paper Ventures.
Uncorrelated Ventures raises $315 million for crypto and software-focused fund
Uncorrelated Ventures has previously backed crypto projects including Compound, Cosmos, dYdX, Helium and Uniswap.
🚀 Project Launches & Updates
Binance launches marketplace for inscription tokens
Binance is taking on its major rival OKX, which introduced similar features earlier this week, to tap the inscriptions market.
Bitfinex Securities launches digital asset services in El Salvador
Following the successful launch of U.S. spot bitcoin ETFs, Bitfinex anticipates high demand for regulated digital asset investment vehicles.
Jupiter token to begin trading on exchanges today
Jupiter’s native token will launch today with subsequent trading on centralized exchanges.
Circle will launch USDC on the Celo network
An imminent governance vote will determine if USDC should become Celo’s official gas currency.
🔥 Other Bits We're Excited About
Bankrupt FTX won't be restarting, but former customers will get money back in full
An FTX lawyer said during a Wednesday hearing that plans for a re-launch of the exchange won’t come to fruition.
Ethereum’s Dencun upgrade goes live on Sepolia testnet
Ethereum core developers deployed the upgrade on the Goerli testnet earlier this month.