Analytics and Data Summit 2018

This talk presents the experience of a customer project where we built a stream-based ingestion into a graph database. It is one thing to load the graph first and then querying it. But it is another story if the data to be added to the graph is constantly streaming in, while querying it. Data is easy to add, if each single message ends up as a new vertex in the graph. But if a message consists of hierarchical information, it most often means creating multiple new vertices as well adding edges to connect this information. What if a node already exists in the graph? Do we create it again or do we rather add edges which link to the existing node? Creating multiple nodes for the same real-life entity is not the best choice, so we have to check for existence first. We end up requiring multiple operations against the graph, which demonstrated to be a bottle neck. This talk presents the implementation of an ingestion pipeline and the design choice we made to improve performance.