Welcome to Lumen!#

Lumen is a fully open-source and extensible agent based framework for chatting with data and for retrieval augmented generation (RAG). The declarative nature of Lumen’s data model make it possible for LLMs to easily generate entire data transformation pipelines, visualizations and other many other types of output. Once generated the data pipelines and visual output can be easily serialized, making it possible to share them, to continue the analysis in a notebook and/or build entire dashboards.

Generate SQL: Generate data pipelines on top of local or remote files, SQL databases or your data lake.
Provide context and embeddings: Give Lumen access to your documents to give the LLM the context it needs.
Visualize your data: Generate everything from charts to powerful data tables or entire dashboards using natural language.
Inspect, validate and edit results: All LLM outputs can easily be inspected for mistakes, refined, and manually edited if needed.
Summarize results and key insights: Have the LLM summarize key results and extract important insights.
Custom analyses, agents and tools: Extend Lumen custom agents, tools, and analyses to generate deep insights tailored to your domain.

Lumen sets itself apart from other agent based frameworks in that it focuses on being fully open and extensible. With powerful internal primitives for expressing complex data transformations the LLM can gain insights into your datasets out-of-the box and can be further tailored with custom agents, analyses and tools to empower even non-programmers to perform complex analyses without having to code. The customization makes it possible to generate any type of output, allow the user and the LLM to perform analyses tailored to your domain and look up additional information and context easily. Since Lumen is built on Panel it can render almost any type of output with little to no effort, ensuring that even the most esoteric usecase is easily possible.

The declarative Lumen data model further sets it apart from other tools, making it easy for LLMs to populate custom components and making it easy for the user to share the results. Entire multi-step data transformation pipelines be they in SQL or Python can easily be captured and used to drive custom visualizations, interactive tables and more. Once generated the declarative nature of the Lumen specification allows them to be shared, reproducing them in a notebook or composing them through a drag-and-drop interface into a dashboard.