DCP: dynamic resource allocation and accounting in a digital marketplace

Automating resource allocation and multi-metric accounting in a federated digital marketplace

Kings Distributed Systems is a Canadian company deploying the Distributed Compute Protocol (DCP), a web platform that aggregates excess computing power from underutilized devices and digital infrastructure and makes it available to researchers and innovators. Their Compute API allows users to trivially express parallel workloads, e.g. Advanced Research Computing, AI/ML, blockchain, mathematical finance. The Protocol automatically distributes those workloads for computation.

Challenge

To address the challenge of providing researchers with sufficient and cost-effective computing resources, Kings Distributed Systems is deploying the Distributed Compute Protocol (DCP), a cross-platform solution that aggregates computing resources from arbitrary devices and digital infrastructure － from smartphones to enterprise servers － and makes it available to researchers and innovators on-demand. DCP would allow both individual institutions as well as federated infrastructures, such as the EOSC, to recapture and allocate underutilized resources, while providing a credit-based accounting system to quantify usage of processing, bandwidth, and storage resources. 

The company holds the vision that the Distributed Compute Protocol becomes the multi-platform standard for distributed and edge computing. Kings Distributed Systems is facilitating access to limitless computing resources to accelerate science, innovation and discovery. Overall, this pilot aims to not only test, but showcase the applicability and value of such a solution for the European Open Science Cloud.

Work Plan

• Instantiate DCP Worker process, a JavaScript program operating on Google’s v8 engine, on EOSC infrastructure. DCP Workers retrieve data and work from the KDS Scheduler, a NodeJS daemon that receives work functions and data sets from the Compute API.
• For an agreed upon term, European and Canadian researchers will deploy physics, astronomy, and mathematical ecology research computing workloads via the Compute API.
• Demonstrate the technical feasibility of distributing computational workloads on EOSC’s digital infrastructure using DCP.
• Monitor and record the value of processor and bandwidth resources consumed on the platform using DCP’s credit-based accounting system. Identify additional metrics that could be accounted for e.g. data storage, other EOSC services.
• Evaluate overall performance and ease-of-use of DCP at both the user (researcher) and infrastructure (EOSC) levels, using feedback to improve the platform.

Partners

Distributed Compute Labs