Amazon recently lost control of IP addresses it uses to host cloud services and took more than three hours to regain control, a lapse that allowed hackers to steal $235,000 in cryptocurrency from users of one of the affected customers, an analysis shows.
The hackers seized control of roughly 256 IP addresses through BGP hijacking, a form of attack that exploits known weaknesses in a core Internet protocol. Short for border gateway protocol, BGP is a technical specification that organizations that route traffic, known as autonomous system networks, use to interoperate with other ASNs. Despite its crucial function in routing wholesale amounts of data across the globe in real time, BGP still largely relies on the Internet equivalent of word of mouth for organizations to track which IP addresses rightfully belong to which ASNs.
A case of mistaken identity
Last month, autonomous system 209243, which belongs to UK-based network operator Quickhost.uk, suddenly began announcing its infrastructure was the proper path for other ASNs to access what’s known as a /24 block of IP addresses belonging to AS16509, one of at least three ASNs operated by Amazon. The hijacked block included 184.108.40.206, an IP address hosting cbridge-prod2.celer.network, a subdomain responsible for serving a critical smart contract user interface for the Celer Bridge cryptocurrency exchange.
On August 17, the attackers used the hijacking to first obtain a TLS certificate for cbridge-prod2.celer.network, since they were able to demonstrate to certificate authority GoGetSSL in Latvia that they had control over the subdomain. With possession of the certificate, the hijackers then hosted their own smart contract on the same domain and waited for visits from people trying to access the real Celer Bridge cbridge-prod2.celer.network page.
In all, the malicious contract drained a total of $234,866.65 from 32 accounts, according to this writeup from the threat intelligence team from Coinbase.
The Coinbase team members explained:
The phishing contract closely resembles the official Celer Bridge contract by mimicking many of its attributes. For any method not explicitly defined in the phishing contract, it implements a proxy structure which forwards calls to the legitimate Celer Bridge contract. The proxied contract is unique to each chain and is configured on initialization. The command below illustrates the contents of the storage slot responsible for the phishing contract’s proxy configuration:
The phishing contract steals users’ funds using two approaches:
- Any tokens approved by phishing victims are drained using a custom method with a 4byte value 0x9c307de6()
- The phishing contract overrides the following methods designed to immediately steal a victim’s tokens:
- send()- used to steal tokens (e.g. USDC)
- sendNative() — used to steal native assets (e.g. ETH)
- addLiquidity()- used to steal tokens (e.g. USDC)
- addNativeLiquidity() — used to steal native assets (e.g. ETH)
Below is a sample reverse engineered snippet which redirects assets to the attacker wallet:
Hello, Amazon? Anybody there?
As shown in the green sections in the upper portion of this graph, which was prepared by Doug Madory of network monitoring company Kentik, the fraudulent BGP announcement began on August 17 at around 19:39 UTC and became globally accepted within seconds. For unknown reasons, the announcement was withdrawn at 20:22, only to return and be withdrawn again three more times over the next 68 minutes.
The announcement claimed that 220.127.116.11/24 originated with AS-14618, one of Amazon’s other ASNs, and should be routed through Quickhostuk’s AS-20943. A new announcement such as this one—which within seconds causes the entire Internet to route Amazon IP addresses through the Quickhostuk ASN—is the kind of event that should have triggered an immediate investigation by the cloud provider. For reasons that remain unknown, Amazon didn’t start announcing the correct path for the /24 block until 23:07 (as indicated in purple in the graph above), more than three hours after the rogue announcements began.
“The key detail here that would have distinguished this alert from the appearance of just another peer of Amazon would have been that the new upstream was seen by 100% of BGP vantage points,” Madory wrote on Thursday. “In other words, this new Amazon prefix was getting exclusively transited by this relatively unknown hosting provider. That should have raised some eyebrows among the Amazon NetOps team.”
Amazon representatives didn’t respond to any of three emails seeking comment for this post sent over the past nine days. Quickhostuk officials also didn’t respond to two emails asking how the fraudulent announcements came to be. This post will be updated if either replies later.
It’s not the first time a BGP attack on an Amazon IP address ended in huge losses of cryptocurrency. In 2018, an uncannily similar event happened to Amazon’s Route 53 domain name system service. The attack caused about $150,000 in cryptocurrency to be drained from account holders with MyEtherWallet.com. The amount stolen probably would have been higher had the attackers obtained a browser-trusted TLS certificate instead of using a self-signed one that required victims to click through a warning.
In the aftermath of the 2018 attack, Amazon added more than 5,000 of its IP prefixes to what are known as ROAs (route origin authorizations), which are public documents stating which ASNs have the authority to announce which IP addresses. The move offered some protection from an RPKI (resource public key infrastructure), which uses digital certificates to connect ASN to their rightful IP addresses.
To bypass the protections, this analysis shows, last month’s attackers added AS-16509 and the more specific /24 route to what’s known as an AS-SET indexed in ALTDB, a free registry for autonomous systems to publish their BGP routing policies.
irrd.log-20220817.gz:31106306-descr: quickhost set
irrd.log-20220817.gz:31106332-members: AS209243, AS16509
irrd.log-20220817.gz:31106392-changed: crussell()quickhostuk net 20220816
irrd.log-20220817.gz:31147656-changed: crussell()quickhostuk net 20220816
An AS-SET is a registration entry that defines the allowable set of ASNs that a provider might encounter from that network.
In analyzing why the move failed to prevent last month’s hijacking, Madory was careful to note that RPKI is intended to limit the impact of inadvertent or accidental hijacks due to routing leaks or misconfigurations, not as a security mechanism to prevent fraud by motivated attackers.
Having said that, it could have still helped if the ROA were set up differently. As it stands, the ROA for the address space in question is quite liberal. Basically, three different Amazon ASNs (16509, 8987, and 14618) can all announce parts of this address space with prefixes ranging in size from a /10 all the way down to a /24. See the output of the Routinator web UI:
An alternative approach to ROA creation would be to do what other networks such as Cloudflare and Comcast have done: set the origin and maximum prefix length to be identical to how the prefix is routed. While this approach incurs an overhead cost of needing to update a ROA every time a route is modified, it also leaves little room for alternate versions of the route to come into circulation.
In fairness, Amazon is hardly the only cloud operator to lose control of its IP addresses in a BGP attack. The vulnerability of BGP to ham-fisted misconfigurations and outright fraud has been evident for more than two decades now. The insecurity is ultimately an industry-wide problem that Amazon alone is powerless to fix.
But as a tier-1 cloud provider that has now suffered at least two BGP hijackings that have cost downstream users money, Amazon should be expected to acknowledge the event, explain why it waited more than three hours to take corrective action, and chart a plan for preventing similar ones in the future. Rather than maintaining radio silence as it has done for the past five weeks, Quickhostuk should also account for this incident and why it appears its ASN issued such a problematic announcement.