The tl;dr of it is there are ways to engineer this so the LLM doesnt get to “make up” data, the LLMs job is to just compose the query, and then it gets run against the DB and that returns to the user directly, preventing the LLM from just making shit up.
MCP Tools are powerful as hell for this, and its actually very viable to do.
This is how you end up with made figures, because the generated query forgot a WHERE clause and no one’s there to check it
See my post for how I’d solve this problem above here: https://lemmy.world/post/46926396/23775592
The tl;dr of it is there are ways to engineer this so the LLM doesnt get to “make up” data, the LLMs job is to just compose the query, and then it gets run against the DB and that returns to the user directly, preventing the LLM from just making shit up.
MCP Tools are powerful as hell for this, and its actually very viable to do.