You Cannot Debug an LLM Feature You Cannot See

A user says the assistant gave a wrong answer. In a normal feature you find the request in the logs. In an LLM feature there were four model calls, a retrieval step, a tool invocation and a summarisation, and your log line says POST /api/chat 200.

Observability for AI features means recording the chain, not the endpoint.

A trace is the unit

from langfuse.decorators import observe

@observe()
def answer_question(question: str):
    docs = retrieve(question)        # nested span
    draft = draft_answer(question, docs)
    return polish(draft)

Each nested call becomes a span with its inputs, outputs, latency and token cost. When the answer is wrong you can see whether retrieval returned the wrong documents, whether the draft was already wrong, or whether polishing dropped a caveat. Those are three completely different fixes, and without the trace you are guessing which one you have.

Prompt versions turn opinions into evidence

Keep prompts in the platform rather than hard-coded, and every trace records which version produced it. "Quality dropped last Tuesday" becomes a diff against a specific change instead of a debate.

It also means a non-engineer can adjust wording without a deployment — useful, but only safe if you have the evaluation set in the next section.

Scores make quality comparable

Attach a score to a trace: a thumbs-up from the user, a rule-based check, or a model grading against a rubric.

langfuse.score(trace_id=trace_id, name="answered_in_arabic", value=1.0)

Now "is the new prompt better" has an answer that survives disagreement. On a bilingual product this is how I catch the classic regression where a change improves English output and quietly degrades Arabic — it never shows up in spot checks because the person checking reads English first.

The privacy problem you must handle

Traces contain full inputs and outputs. If users type personal data into your product, you have just built a second database of it, and it needs the same retention policy, access control and deletion path as the first.

Redact before it leaves the application, mask fields you do not need, and set retention deliberately. Self-hosting is available and is the right default for anything regulated.

The minimum worth having

Even without evaluations, having traces at all changes the work. "It gave a bad answer" stops being unactionable and becomes a specific span with specific inputs. That alone justifies the afternoon it takes to instrument.

Resources

• Repo: langfuse/langfuse
• Docs: langfuse.com/docs
• Video walkthroughs: YouTube: langfuse llm observability setup
• Related: One gateway in front of every model