As the scale of available data continues to grow, so does the need for scalable and intelligent data processing systems to swiftly harness useful knowledge. Especially in high-stakes domains such as life sciences and finance, alongside scalability, transparency of data-driven processes becomes paramount to ensure the utmost trustworthiness.
Started by scientists coming from the Knowledge Graph labs across the University of Oxford and TU Wien, Prometheux, an NVIDIA Inception company, builds AI capable of explaining its exact logical process. From drug repurposing for AstraZeneca to financial data processing with the Applied Research Team of the Central Bank of Italy, Prometheux technology powers highly scalable and explainable reasoning over some of the world’s largest knowledge graphs.
Prometheux has seamlessly integrated the RAPIDS accelerator for Apache Spark into their proprietary knowledge graph management system, Vadalog Parallel, to leverage NVIDIA GPUs. They achieve significant speedups and cost savings for their clients when processing large knowledge graphs with hundreds of millions of entities and billions of relations.
Knowledge graphs and reasoning
Over the last decades, due to the ever-growing scale of available data, there has been a rapid rise in the popularity of large enterprise knowledge graphs. And a corresponding rise in the scalable and intelligent processing systems with which to exploit them.
Knowledge graphs serve as a backbone for data integration and provide a common representation structure to enable query answering across large data sources. A knowledge graph can be defined as a semi-structured data model composed of the following:
- An extensional component: Existing entities and relations integrating knowledge from heterogeneous data sources.
- An intensional component: Domain knowledge, either in the form of statistical and ML models or defined declaratively with logical rules.
- A derived extensional component: Produced through the application of domain knowledge to the extensional component in the so-called reasoning process.
From intricate company ownership graphs to protein interaction networks, knowledge graphs provide a concise and intuitive abstraction for a variety of domains and can be used to model real-world entities and their interrelations.
Vadalog Parallel
Resembling the interplay of intuition and logical thinking in human intelligence, AI is also moving towards neurosymbolic, a synergistic combination of ML and logic-based reasoning.
Building towards neurosymbolic AI, Prometheux provides Vadalog Parallel, its knowledge graph management system (KGMS) that offers a deductive fully explainable framework. This KGMS combines data with domain logic and automates complex reasoning tasks over large knowledge graphs with high scalability and transparency at its core.
Vadalog Parallel acts as middleware between heterogeneous enterprise data sources and applications on top of them. It serves as a backbone for data integration without data migration. Encoding domain logic at a high level enables the rapid development of entirely new solutions without having to program extensive code or design algorithms.
Vadalog Parallel offers database-agnostic compatibility, seamlessly connecting to all major databases (RDBMS, RDF, and NoSQL, for example, Neo4j and Mongo, and so on), as well as diverse data sources (CSV, Parquet, JSON, and so on). It enables modeling and reasoning over graphs, hypergraphs, and simplicial complexes without any constraints on the arity, whether dealing with tuples, triples, or n-tuples.
Domain logic is encoded declaratively at a high level, enabling domain scientists to directly extract insights from large knowledge graphs in an automated, explainable, and intuitive logical manner, saving time and computational resources.
For each task, Vadalog Parallel natively provides the step-by-step, logical full explanation of the reasoning process. Up to billions of logical steps are computed at lightning speed, accompanied by a compact visual explanation to enable faster interactions with domain experts.
Thanks to its expressive framework and its distributed processing, Vadalog Parallel achieves low computational complexity and scalability in practice and enables modeling complex domains with advanced data analysis features.
Indeed, it enables efficient graph traversal and captures both regular path queries (for navigating graphs using pattern matching, for example, Cypher), and SPARQL under the OWL2 QL regime (for querying the semantic web). It supports counterfactual and temporal reasoning, full recursion, and existential quantification, unlike other state-of-the-art systems, and outperforms existing big data analytics tools.
With the unprecedented scale of datasets that we encounter today, new requirements on the scalability and flexibility of reasoning engines have emerged so that AI techniques can execute sufficiently fast. To guarantee such desiderata, Prometheux precisely studies and develops compilation techniques that shift classical reasoning, inference methodologies, and tools onto Big Data platforms.
Vadalog Parallel already achieves unmatched scalability even when operating solely on CPUs. However, when reasoning over some of the world’s largest knowledge graphs (with hundreds of millions of entities and billions of relations), GPUs become an essential asset.
Explainable drug repurposing and beyond
Vadalog Parallel finds diverse applications in the field of computational biology, including its proven tangible impact on indication expansion.
Prometheux enables life sciences organizations to scalably reason over large biological (and other kinds of) knowledge graphs in an automated and logically explainable manner. This enables more informed decisions and faster interactions with domain experts, resulting in accelerated pipeline development globally towards trustworthy precision medicine.
Through the automated analysis of both proprietary and client datasets, Prometheux unravels hidden insights through logical reasoning and unlocks the potential of existing drugs for new therapeutic purposes. By computing billions of logical steps at lightning speed, Vadalog Parallel produces hundreds of successfully validated indications (and many more to study), effectively providing a dynamic and explainable recommendation system for indication expansion.
The compact visual explanation of the recommendations enables faster interactions with domain experts. It leads to rapid adaptation to feedback bringing everyone from initial knowledge to novel indications in a fraction of the time of traditional techniques.
In the following sections, we also provide an experimental analysis of Vadalog Parallel with NVIDIA GPUs for a series of reasoning tasks. We use Prometheux’s internal pilot biological knowledge graph (BIO KG), a starting point for life sciences organizations that have yet to build their own knowledge graphs.
BIO KG has ~4.7M data points spanning compounds, diseases, genes, biological pathways, symptoms, and more. Vadalog Parallel achieves up to 9x speedups with Spark RAPIDS.
Financial institutions
Another compelling application of Vadalog Parallel unfolds within the dynamic realms of finance and economics. Prometheux’s knowledge graph-aided approach empowers an enhanced understanding of intricate interconnections between financial entities, whether they are institutions, companies, financial intermediaries, other types of shareholders, or transactions.
It can be employed for FinTech, RegTech, SupTech, and InsurTech applications encoding international regulations, as well as other domain logic. The goal is to automatically reason over knowledge graphs and achieve AI-aided banking supervision, compliance checks, creditworthiness evaluation, anti-money laundering, fraud detection, shock propagation, company control, detection of takeovers, and more.
This comprehensive approach empowers analysts to proactively manage risks, respond to challenges, optimize strategies, and foster financial stability.
Also for this domain, we show Vadalog Parallel performances with NVIDIA GPUs when reasoning over a company ownership graph (Company KG).
Company KG is a synthetically constructed knowledge graph reflecting the well-known topology of Italian companies, counting 8M ownership edges between companies and shareholders. On such graphs, Vadalog Parallel achieves up to 3x speedups with Spark RAPIDS.
Strategy and solution design
The Vadalog Parallel architecture has the following key components to execute reasoning tasks efficiently on top of distributed frameworks such as Spark, Flink, GraphX, and more:
- Rule parsing
- Logic optimization
- Query planning
- Planner optimization
- Query compilation
Overall, Vadalog Parallel exposes a reasoning API with the following interface:
reason(kg_ref,domain_logic)
A client application issues calls to the reasoning API specifying a reference to a knowledge graph kg_ref in which it activates the reasoning process. Vadalog Parallel connects to and handles repositories of knowledge graphs, each with unique identifiers kg_ref. The reasoning engine encodes domain_logic into a set of distributed operations (narrow, wide transformations, that is, shuffling), and computes the answer to the reasoning task. It either expands the knowledge graph with new knowledge or materializes the output in the specified output data sources.
To ensure even faster and reliable processing, Prometheux has seamlessly integrated Spark-RAPIDS into Vadalog Parallel going beyond traditional Spark capabilities.
The RAPIDS Shuffle Manager, an integral part of Spark-rapids, provides a significant advantage by introducing custom mechanisms for exchanging shuffle data. This innovation offers two distinct modes of operation: Multi-Threaded and UCX, which can be configured to leverage GPU-to-GPU communication and RDMA capabilities. It unlocks unmatched levels of performance and efficiency for the reasoning tasks with Vadalog Parallel.
Scenarios and data
Navigating graphs can be very challenging without powerful recursion. Standard SQL, for instance, lacks native support for recursion.
The Vadalog Parallel framework enables full recursion. In this post, we show how both graph traversal and graph analytics tasks are significantly sped up by integrating GPUs through Spark-RAPIDS.
We categorized the tasks into the following distinct types:
- Non-recursive: The execution plan for the set of operations is structured as a tree and each distributed operation is executed one time.
- Recursive: The execution plan for the set of operations is structured as a graph and the set of distributed operations is applied until the knowledge graph has been exhaustively explored.
In doing so, we provide the first assessment of NVIDIA Spark-RAPIDS with recursive operations.
Experimental results
Figure 3 shows the four knowledge graph analytics tasks containing recursive and non-recursive operations, two over the Bio KG and two over the Company KG. For all measurements, we ran each single experiment 10 times and averaged the results.
Figure 3 shows four reasoning tasks over the knowledge graphs introduced earlier. We provide the relevant experimental evaluation with particular attention to the speedups achieved with NVIDIA GPUs and Spark RAPIDS.
Hardware and software configuration
In all the experimental analyses, Vadalog Parallel was executed on top of a Spark 3.3.2 standalone cluster integrated with Spark-RAPIDS v23.08.1, with CUDA v12.0 and Java v8 as Spark language. The cluster was locally installed on an Amazon EC2 AMI p3.16xlarge with 64 vCPU 8 GPU, 480 GB RAM, and eight NVIDIA V100 Tensor Cores, each having 16 GB of GPU core memory.
Test setup
All the tasks were required to perform the following steps:
- Call the Vadalog Parallel reasoning API passing in input the domain logic describing the task and the reference to one of the two knowledge graphs.
- Establish the connection with the Spark cluster.
- Extract the input subgraph from the specific knowledge graph.
- Execute the task.
- Write the output in the Parquet file.
| KG | Edges | Task | Description | Operations | Reasoning times (sec) |
BIO KG | ~4.7M | Compound/Gene/Disease Similarity | Determine the pairwise similarity for genes, compounds, and diseases based on common features | Non-recursive Wide Transformations: 3 Joins, 3 Aggregations; Non-recursive Narrow Transformations: 3 Maps, 3 Filters; | CPU: 386 GPU: 43 |
BIO KG | ~4.7M | Guilt by Association | Recommend new indications for each compound based on similar ones treating a set of similar diseases | Non-recursive Wide Transformations: 11 Joins, 11 Aggregations; Non-recursive Narrow Transformations: 52 Maps, 26 Filters; | CPU: 520 GPU: 220 |
Company KG | ~8M | All Company-To-Company Links | Determine the pairwise connectivity between all nodes | Non-recursive Wide Transformations: 1 Aggregation; Non-recursive Narrow Transformations: 2 Maps, 1 Filters; Recursive Wide Transformations: 1 Join, 2 Aggregations; Recursive Narrow Transformations: 4 Maps, 4 Filters | CPU: 141 GPU: 46 |
Company KG | ~8M | Company Control | Find all pairs of controllers for each company | Non-recursive Wide Transformations: 2 Aggregations; Non-recursive Narrow Transformations: 2 Maps; Recursive Wide Transformations: 1 Join, 1 Aggregation; Recursive Narrow Transformations: 1 Map | CPU: 94 GPU: 36 |
Conclusion
In this post, we discussed the rapid rise in the popularity of reasoning with logic over large enterprise knowledge graphs as well as the scalable and intelligent processing systems to exploit them. We showcased Vadalog Parallel, Prometheux’s knowledge graph management system. Vadalog Parallel is a powerful framework to combine data with domain logic and automate complex reasoning tasks. It’s a solution advancing us toward neurosymbolic AI, a synergistic combination of ML and logic-based reasoning.
We also discussed the application of Vadalog Parallel in the financial and life sciences realm. The integration of RAPIDS enables significant speedups and cost savings leveraging NVIDIA GPUs when processing some of the world’s largest knowledge graphs.
For more information about Prometheux, contact them via email. For Spark 3.0 and RAPIDS, see the RAPIDS developer forum.
