Data Science Wire

Compiling mixed-format source data into one linked, provenance-tracked artifact for AI agents

Reddit r/dataengineering1mo4 min read

I've been building an open-source tool that takes a bunch of mixed data (PDFs, spreadsheets, decks, recordings, exports, etc.) and compiles it into a single JSON artifact: a graph of nodes and edges where every fact keeps a reference back to the exact source span it came from. Extraction runs per-modality instead of as one generic text pass. Spreadsheets get profiled into a schema (dimensions/measures) rather than dumped as cells, PDFs go through text and table extraction, recordings get transcribed, and so on. After that it links across sources into one graph and tags each fact by fidelity: c

Read the full story at Reddit r/dataengineering

More in MLOps / LLMOps