In this post, I’ll walk you through the best LLM observability tools available today—including both open source projects and enterprise platforms—so you can keep your AI apps reliable, efficient, and compliant (and maybe even help you ace those SEO clicks).

What is LLM Observability, and Why Does It Matter?

LLM observability means tracking, evaluating, and improving how language models perform in the real world. Whether you’re building a chatbot, a content generator, or mission-critical automation, observability tools help answer questions such as:

• Why did my AI output something weird?
• How much is this costing me?
• Can I catch hallucinations before my users do?
• Is my prompt engineering actually improving things?

Without proper observability, issues like hallucinations, latency spikes, or costly inefficiencies go unnoticed—hurting both trust and the bottom line.

Top Open Source LLM Observability Tools

Love tinkering or want full control? Here’s a curated list of the best open source LLM observability platforms that you can host yourself or tweak for your needs:

Proprietary \& Enterprise LLM Observability Platforms

Prefer something more plug-and-play with official support? Check out these leading managed and commercial solutions:

• Arize AI (Phoenix core): Unified monitoring, tracing, evaluation, supports most frameworks
• LangSmith (by LangChain): Deep observability for LangChain workflows
• Galileo AI: Real-time tracing and notification flows
• Datadog: Enterprise monitoring, new LLM features for OpenAI and LangChain users
• HoneyHive: End-to-end evals and monitoring
• Future AGI: Real-time anomaly detection, alerts, evaluation integrations
• Weights \& Biases (Weave): LLM pipeline tracing, prompt logs, metrics

Tip: Many of these vendors offer free or community tiers if you’re just experimenting.

Specialized \& Niche Tools Worth Knowing

• AgentOps, CrewAI: Multi-agent tracing for complex workflow apps (mix of open and closed source)
• MLflow: Traditional ML monitoring, with new LLM add-ons
• DeepEval, Confident AI: LLM quality testing and evaluation
• Aporia, WhyLabs, LangKit: General ML observability tools now supporting LLM workflows
• LlamaIndex Observability: Built-in tools for RAG and document Q\&A frameworks

Quick Comparison Table: Open Source Leaders

Key Features to Watch For

• Tracing: Visualize request/response flows, spot bottlenecks.
• Prompt Management: Version control, A/B testing, and playgrounds.
• Evaluations: Automated and human-in-the-loop scoring for quality, relevance, hallucinations, etc.
• Cost/Token Monitoring: Track cost and token usage to rein in experiment budgets.
• Framework Integrations: Plug into your existing LangChain, LlamaIndex, or RAG stack.
• Self-Hosting: Most open source tools support on-prem installs—crucial for sensitive data!

Final Thoughts: Choosing the Best Tool for Your Needs

The right LLM observability stack depends on what you’re building:

• OpenLatency tools (like OpenLLMetry, SigNoz) are perfect for enterprises running Kubernetes or with established observability pipelines.
• Self-hosters and startups should check out Langfuse, Helicone, or PostHog for robust features at zero cost.
• Production teams needing support or deep evals might benefit from LangSmith or Arize AI.

With the LLM tooling ecosystem growing rapidly, there’s never been a better time to experiment, ship faster, and keep your users—and your CFO—happy.

Got a favorite LLM observability tool I missed? Drop a comment or send a tweet—let’s keep this guide up to date!

The Problem

Let's say you have a CompanyStructure model with a ManyToMany relationship to Django's CustomUser. By default, Django admin gives you a nice horizontal filter widget on the CompanyStructure side, but not on the CustomUser side. This creates an inconsistent user experience when managing these relationships.

The Solution

After some digging into Django's internals, I found an elegant solution using Django's RelatedFieldWidgetWrapper and a custom ModelForm. Here's how to implement it:

1. First, let's look at our models:

from django.contrib.auth.models import AbstractUser
from django.db import models
import treenode.models as treenode_models

class CustomUser(AbstractUser):
    objects = CustomUserManager()

    def __str__(self):
        return self.username

class CompanyStructure(treenode_models.TreeNodeModel):
    name = models.CharField(max_length=255)
    users = models.ManyToManyField(
        CustomUser, 
        related_name='company_structures', 
        blank=True
    )
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.name

2. The Magic: Custom Form with RelatedFieldWidgetWrapper

Here's where the magic happens. We'll create a custom form that adds the horizontal filter widget:

from django import forms
from django.contrib import admin
from django.db import models
from django.contrib.admin import widgets

class CustomUserForm(forms.ModelForm):
    company_structures = forms.ModelMultipleChoiceField(
        queryset=CompanyStructure.objects.all(),
        widget=widgets.RelatedFieldWidgetWrapper(
            widget=widgets.FilteredSelectMultiple('company structures', False),
            rel=models.ManyToManyRel(
                field=CustomUser,
                to=CompanyStructure,
                through=CustomUser.company_structures.through
            ),
            admin_site=admin.site
        )
    )

    class Meta:
        model = CustomUser
        fields = '__all__'

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # Initialize with existing relations
        if self.instance and self.instance.pk:
            self.fields['company_structures'].initial = (
                self.instance.company_structures.all()
            )

    def save(self, commit=True):
        instance = super().save(commit=False)
        if commit:
            instance.save()
        # Save the ManyToMany relations
        instance.company_structures.set(self.cleaned_data['company_structures'])
        return instance

3. Register with Admin

Finally, we hook it all up in the admin:

@admin.register(CustomUser)
class CustomUserAdmin(UserAdmin):
    form = CustomUserForm
    # ... other admin configurations ...

See how it looks

How It Works

Let's break down the key components that make this solution work:

1. RelatedFieldWidgetWrapper: This is the secret sauce. It's the same widget Django uses internally for related fields, but we're explicitly implementing it for our reverse relationship.

2. FilteredSelectMultiple: This creates the actual horizontal filter interface with search functionality.

3. ManyToManyRel: This tells Django about the relationship type and provides necessary metadata for the widget to function properly.

Key Benefits

• Consistent UI: Users get the same filtering experience from both sides of the relationship

• Better UX: The horizontal filter widget is more user-friendly than a basic select field

• Search Functionality: Built-in search makes it easier to find specific items in large datasets

Potential Gotchas

1. Performance: With large datasets, you might want to optimize the queryset in the ModelMultipleChoiceField

2. Form Initialization: Make sure to handle the initial values correctly in the __init__ method

3. Saving: Don't forget to implement the save method to handle the M2M relationship properly

Attention: Django 5.1+

Since we override the form class, we might want to bring back this beauty

Complete Code: https://gist.github.com/Kenan7/0a5169242f13b95acd362e5d7a841607

In Conclusion

This solution provides a clean, symmetrical interface for managing ManyToMany relationships in Django admin. While it requires a bit more code than the default setup, the improved user experience makes it worth the effort.

Remember to adapt the code to your specific needs and consider adding any necessary validation or customization to match your project's requirements.

What is Django Middleware?

Middleware in Django is a framework of hooks into Django's request/response processing. It's a lightweight, low-level plugin system for globally altering Django's input or output.

Each middleware component is responsible for doing some specific function. For example, Django includes a middleware that adds the CSRF token to responses and another that handles user sessions.

How Does Middleware Work?

Middleware classes are called in the order they're defined in the MIDDLEWARE setting. During the request phase, Django calls the process_request() method of each middleware in order. During the response phase, the process_response() methods are called in reverse order.

Here's a simplified view of the request/response cycle:

1. Request comes in

2. Middleware 1 processes request

3. Middleware 2 processes request

4. View processes request and produces response

5. Middleware 2 processes response

6. Middleware 1 processes response

7. Response goes out

Creating Custom Middleware

Let's look at some examples of custom middleware:

Example 1: Timing Middleware

This middleware will measure how long each request takes to process:

import time

class TimingMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        start_time = time.time()

        response = self.get_response(request)

        duration = time.time() - start_time
        response['X-Page-Generation-Duration-ms'] = int(duration * 1000)
        return response

This middleware adds an X-Page-Generation-Duration-ms header to each response, showing how many milliseconds the request took to process.

Example 2: IP Restriction Middleware

This middleware will restrict access to certain views based on IP address:

from django.http import HttpResponseForbidden

class IPRestrictionMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        allowed_ips = ['192.168.1.1', '192.168.1.2']
        ip = request.META.get('REMOTE_ADDR')

        if request.path.startswith('/admin/') and ip not in allowed_ips:
            return HttpResponseForbidden("You are not allowed to access this resource.")

        return self.get_response(request)

This middleware checks if the request is for the admin area and if so, verifies that the client's IP is in the allowed list.

Example 3: Request Logging Middleware

This middleware will log details about each request:

import logging

logger = logging.getLogger(__name__)

class RequestLoggingMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        logger.info(f"Request: {request.method} {request.path} from {request.META.get('REMOTE_ADDR')}")

        response = self.get_response(request)

        logger.info(f"Response: {response.status_code}")
        return response

This middleware logs the HTTP method, path, and IP address for each request, as well as the status code of each response.

Using Your Middleware

To use your custom middleware, add it to the MIDDLEWARE setting in your Django settings file:

MIDDLEWARE = [
    # ... other middleware classes ...
    'path.to.TimingMiddleware',
    'path.to.IPRestrictionMiddleware',
    'path.to.RequestLoggingMiddleware',
]

Remember, the order matters! Middleware classes are processed in the order they appear in this list.

Conclusion

Middleware is a powerful feature in Django that allows you to process requests and responses globally. Whether you're adding security features, logging information, or modifying responses, middleware provides a clean and reusable way to add functionality to your Django application.

By creating custom middleware, you can keep your views clean and focused on their primary logic, while handling cross-cutting concerns at the application level.

After researching a little more I finally found a solution that works perfectly. Proxy.py

This awesome tool is written in python, has docker configurations, and HTTPS certificate generator.

So let's start.

I reckon that you are already on your server's terminal, you have git and python3-virtualenv installed.

git clone https://github.com/abhinavsingh/proxy.py.git

cd proxy.py

python3 -m venv env
. env/bin/activate

pip install -r requirements.txt

So if you run all commands so far, you should be good.

Now, this is where we do a little customization. In proxy.py docs, they are building and running containers through docker engine, but we will create our own docker-compose file for more comfort.

if you don't have already installed docker and docker-compose please follow these links:

for docker: docker/download

for docker-compose (ubuntu): linuxize/docker-compose

Let's dig in.

create a file called docker-compose.yml in our cloned directory (proxy.py)

nano docker-compose.yml

paste this context

version: "3"

services:
  proxy:
    build: .
    container_name: custom-proxy
    ports:
      - "8899:8899"
    restart: always

if you have an entry for hosts file you can use extra-hosts

here is sample

services:
  proxy:
    build: .
    container_name: custom-proxy
    ports:
      - "8899:8899"
    restart: always
    extra_hosts:
      - "example.com:1.2.3.4"

Let me break these lines for you

proxy is just the name of the service

build is where our Dockerfile is located (yes it's in the current directory!)

container_name is not necessary, I just did it because I like it that way

ports: we expose the ports to the public (proxy.py default is 8899)

restart: always is simply saying start running container again if an error occurs

That's that.

Now that we have our docker-compose file we can update and run our containers more easily! Run the following command to build and get your container up.

sudo docker-compose build

when this command finishes you'll have something like this

...
Step 14/15 : ENTRYPOINT [ "proxy" ]
 ---> Using cache
 ---> a18dd7a78abb
Step 15/15 : CMD [ "--hostname=0.0.0.0" ]
 ---> Using cache
 ---> 012db77b2da2
Successfully built 012db77b2da2
Successfully tagged proxypy_proxy:latest

to get you container up and running

sudo docker-compose up

note: add -d parameter at the end to run in the background

Creating exbir-proxy ... done
Attaching to exbir-proxy
exbir-proxy | 2020-12-04 18:43:35,267 - pid:1 [I] load_plugins:334 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
exbir-proxy | 2020-12-04 18:43:35,268 - pid:1 [I] listen:113 - Listening on 0.0.0.0:8899
exbir-proxy | 2020-12-04 18:43:35,277 - pid:1 [I] start_workers:136 - Started 2 workers

CONGRATULATIONS! your proxy works in your server on port 8899!

At this point, I have a little bonus for you. If you have your own customizations too, to make docker-compose controls easier, go ahead and make temporary alias for yourself.

alias dd="sudo docker-compose down -v && sudo docker-compose build && sudo docker-compose up -d"

You can add it to your .bashrc (or other shells) file if you want to make it permanent.

Please ask me any questions related to this post!