Smartening the IPFS DHT
14-12-2020
One of the main content routing systems of the IPFS architecture is a Distributed Hash Table (DHT). DHTs are well-studied overlay networks that allow key-based lookup in a fully decentralized manner. IPFS leverages a DHT system, implemented using the Kademlia protocol, to provide content lookup: all IPFS nodes "advertise" content items they store in the DHT and this results in a distributed "dictionary" used for looking up content.
Content advertisements and lookups are routed through an overlay network composed of the nodes participating in the DHT. Currently, routing in the overlay network is only based on node identifiers, which are flat, randomly generated numbers. This results in significant "path stretching," i.e., an overlay path between two nodes can be considerably longer than the corresponding underlay IP path. Additionally, IPFS’s DHT is composed of user-managed nodes, i.e., it does not rely on a centralized or single-entity managed infrastructure. Although this gives IPFS intriguing scalability, decentralization, robustness, survivability, security, and privacy properties, it affects network availability since user nodes often experience high "churn" rates, frequently disconnecting from the network. Protocol Labs recently joined forces with the Mobile Multimedia Laboratory at the Athens University of Economics and Business in a project that aims to solve these problems by designing and applying a multilevel DHT design.
The proposed solution will apply the Canon approach in IPFS’s DHT. Canon is a framework for building DHTs that allow node clustering based on the underlay network topology. The novelty of Canon is that it does not propose a new DHT design, instead it builds on existing DHT systems by modifying their routing algorithms. Canon belongs to a class of DHTs known as "horizontal multi-level" DHTs. DHTs of this category have the advantage of not relying on centralized, special-purpose, powerful "super nodes" (also known as super peers). Instead, they equally distribute load across all nodes. Furthermore, horizontal multi-level DHTs do not impose any requirements on node identifiers (as opposed to, for example, "vertical" DHTs, which typically require node identifiers to reflect node hierarchy). This allows flat, random identifiers that can be mapped to public keys, hence offering better security. A multilevel DHT built using Canon has two important performance and security enhancing properties: path locality and convergence of paths exiting a cluster.
The path locality property guarantees that a path between two nodes belonging to the same cluster never leaves that cluster. If node clustering is defined using a network-related metric (e.g., latency) or condition (e.g., geo-location), nodes belonging to the same network domain form a cluster. In that case, path locality results in significant performance and availability improvements: paths between nodes of the same domain will never exit that domain, and, no matter the churn of the external nodes, as long as the nodes of the domain are stable, intra-domain paths will never fail. Similarly, if clustering is defined using a trust metric, path locality increases security since a "local" path will include only equally trusted nodes.
Convergence of paths exiting the cluster guarantees that all paths moving from inside a cluster toward a given destination outside that cluster will always exit the cluster through the same node. This property creates great opportunities for caching as well as for aggregation of requests and responses. Furthermore, these exit nodes are ideal locations for implementing security policies. It should be noted that these exit nodes are not ultimately single points of failure since, if they fail, their role will be automatically assigned to another node of the cluster.
Our joint project, which started in July 2020 with an expected duration of 15 months, is composed of two main phases. In the first phase, we will experiment with various clustering algorithms, bootstrap mechanisms, and deployment strategies through simulations. To this end, we will rely on PeerNet, an event-based P2P simulator extending and relying on the popular PeerSim simulator. During this phase, we will use the simulator to evaluate the two multilevel DHTs resulting from applying the Canon framework to Pastry and Kademlia. Furthermore, we will adapt IPFS protocols to take advantage of the hierarchical DHT properties.
During the second phase of the project, we will extend the libp2p library, which underpins IPFS and related systems, to include a multi-level DHT implementation, as well as auxiliary tools. Our implementation, which will be made available as a free, open-source project, will be used for experimenting with the IPFS protocol stack and for designing a hassle-free migration strategy.
We expect this to be a fun, promising project, and we anticipate that it will contribute to the improvement of the performance and security of IPFS. We also anticipate that a multi-level DHT design will be able to support novel applications by providing service-level guarantees.