LLMs have limited-size “context windows” that cap the amount of input they can work with. LLM performance degrades rapidly on more challenging tasks as more input is added. This makes their “effective context windows” much smaller in certain cases, including knowledge graph generation. Regardless of the exact input cutoff, context windows introduce a problem when using LLMs to generate graphs, where the graph generated for one set of text added to a context window (commonly called a “chunk”) conflicts with the graph generated from another chunk.

All LLMs, and most frameworks built on top of them to generate graphs, suffer from this problem. Aside from Unize ST-0.5, there is one other system we know of that attempts to address this challenge, LangChain’s LLMGraphTransformer module. It was created by the AI team at Neo4j, the leading graph database company.

Perhaps due to the previous impossibility of larger-scale graph generation, we were unable to find suitable benchmarks, so we created our own called KGStorage with a single-input dataset of 5,000 characters and a multi-input dataset of 20,000 characters.

We ran Unize ST-0.5, LangChain’s LLMGraphTransformer, and GPT-4o (selected to represent unmodified LLM output) against KGStorage with the following results:

Unize ST-0.5 achieves state-of-the-art results on KGStorage’s multi-input dataset, topping critical measures like node accuracy by wide margins. These correctness metrics indicate that most nodes, relationships, and associated data accurately reflect the source text. LangChain’s performance is much closer to Unize ST-0.5 on KGStorage’s single-input dataset, achieving 89% node accuracy compared to Unize ST-0.5’s 96%, while GPT-4o scores 43%. This indicates LangChain provides value on top of raw LLM output on smaller inputs, though its performance declines significantly on larger inputs. You can read our full results and methodology here.

While Unize ST-0.5’s performance results are strong, please keep in mind it’s an alpha, and performance is not yet close to human level. Improvements are actively in the works though!

You can sign up for Unize Storage here. Our user-friendly playground allows developers and non-developers to easily experiment with the system, and we are currently giving away free API credits while supplies last. Please get in touch if you have any thoughts or feedback!