Django: Grappelli Admin Dashboard Graphs

Created: 07.07.2022 | Last edited: 21.07.2022

Introduction

Motivation

Orignally I wanted to "one day" add a few cool graphs and charts to my django admin dashboard.

However, I needed a file browser solution and settled with django-filebrowser. This requires django-grappelli. I had no idea of what grappelli was. But now that I had it of course I had to play around with it, resulting in my "add graphs to django admin"-project coming to life way earlier than I thought.

What we will build today:

We will create an admin dashboard that overwrites our default dashboard using grappelli and add graphs to it displaying aggregated data from our database or a website analytics API; looking like this:

grappelli Dashboard with custom graph module displaying two charts

What is django-grappelli?

So basically grappelli overwrites the default django admin and especially the admin landing page, calling it a "dashboard" for now. Besides a few more elements and changed CSS there isn't much to see. You can generate a dashboard.py file which allows you to define elements on your admin landing page and rearrange them using python. So basically a python website block builder for your admin page. However, it doesn't offer as many new features as I would have liked or hoped for. But what it does well is to provide you with an easy interface to inject your own HTML elements and functions inside the admin dashboard. That on it's own is pretty great as it can be quite a hassle to change the django admin itself. You would need to overload, overwrite, or recreate your own and it just causes a lot of overhead work if there is just a simple thing you wanted to achieve. Also, I like how it keeps the way the django admin itself works pretty much intact, allowing you to use all your knowledge about customizing stock django admin views on top.

So how to add graphs?

As mentioned, there aren't too many prebuild components (grappelli calls them "modules" - now this won't get confusing...). That's why we need to build our own if we want to display graphs and charts. However, the thing I like the least about grappelli is its documentation. And of course, you won't really find much about building your own modules there. Luckily, reverse engineering existing grappelli modules isn't to hard to build our own. Since we build the whole graphing module from scratch we can use any technology we want. We could use fancy JavaScript libraries but...I am a Data Scientist. So I know Python much better and since Django is written in Python as well...I feel like it makes more sense to implement Python plotting libraries (even if they just offer a Python API and are written in C). Let's set the goals:

Setting goals for this project

Alright, so what do we even want to do?

  • Build a custom, reusable, standalone grappelli module for our admin dashboard
  • This module should:
    • Use matplotlib and or seaborn (which is based on matplotlib anyway)
    • Be easy to use
    • Let us customize our graphs as extensively as matplotlib or seaborn offer themselves without the need to touch the code of the module to do so
  • Display our graphs on our admin page using django ORM data

Sounds cool? To me it does.

Preparation

Basic grappelli setup

So if you haven't used grappelli before like me initially, let's set it up really quick: pip3 install django-grappelli, add it to our INSTALLED_APPS (as well as django.contrib.admin AFTER grappelli if you haven't already), and we are already halfway done.

We need to add its URLs to our URLs like so:

from django.conf.urls import include

urlpatterns = [
    path('grappelli/', include('grappelli.urls')),
    path('admin/', admin.site.urls), # don't forget your admin urls - they are still needed!
]

Next, we need grappelli to generate a dashboard file for our site. It injects its commands into manage.py so we can just run:

python3 manage.py customdashboard

Note: you will also need to add 'django.template.context_processors.request' to your templates context processors; but this should be a default setting.

To configure grappelli a little bit further let's add the following to our django settings file:

# grappelli dashboard setup
GRAPPELLI_INDEX_DASHBOARD = 'YOURSITE.dashboard.CustomIndexDashboard'  # give Grappelli the path to our generated dashboard
GRAPPELLI_ADMIN_TITLE = 'Amazing Admin Dashboard :)'  # optional: set a title it should display on our dashboard

# this is NOT a Grappelli setting:
GRAPPELLI_CUSTOM_TEMPLATES = os.path.join(BASE_DIR, 'templates/dashboard')
# I came up with this variable as we need it later on.
# this variable should basically include your path to your templates folder where you want our custom Grappelli module html templates to live

We should be done with our basic setup and settings by now. If you execute collectstatic and run your server you should see your new admin dashboard on the same URL where your old admin page used to be.

Preparing the customdashboard Layout

Building our custom Graphs-module

Setting a file structure

Building a new custom Template for the Graphs-Module

We will start by building an HTML-template for our new Graphs-module. This might sound like an odd order to do things, starting with the "view"-part here. But looking into how grappelli modules work this will be important for our future class.

So let's take a look inside the grappelli files and pick a random template for an existing module. I am just randomly picking link_list.html which looks like this:

{% extends "grappelli/dashboard/module.html" %}
{% load i18n %}
{% block module_content %}
    <ul class="grp-listing-small">
        {% spaceless %}
            {% for child in module.children %}
                <li class="grp-row">
                    <a class="{% if child.external %}grp-link-external{% else %}grp-link-internal{% endif %}" href="{{ child.url }}" {% if child.description %} title="{{ child.description }}"{% endif %}{%if child.target %} target="{{ child.target }}"{% endif %}>{{ child.title }}</a>
                </li>
            {% endfor %}
        {% endspaceless %}
    </ul>
{% endblock %}

As we can see besides the base structure and some standard tags, grappelli DashboardModule-classes usually get us a module.children variable into the context of our template. However, we simply want to display a graph. The idea here will be to generate the graph as SVG, clean the SVG string, and simply render the HTML SVG tag out inside the template. Overall we need the module.children variable to only contain a string which we will render out as HTML.

So inside the folder where we want to store our custom module templates (in my case: templates->dashboard->modules) we can simply add a new file graph.html and copy-paste the contents shown before. Next, we only need to strip the for loop as our module.children will be a simple string instead of a list or dictionary, and add a "safe" filter onto it to make sure the HTML inside our passed string gets rendered as actual HTML. Our resulting custom graph template file ends up like this:

{% extends "grappelli/dashboard/module.html" %}
{% load i18n %}
{% block module_content %}
    <ul class="grp-listing-small">
        {% spaceless %}
                <li class="grp-row">
                    <p class="grp-font-color-quiet">{{ module.children|safe }}</p>
                </li>
        {% endspaceless %}
    </ul>
{% endblock %}

With the template done we now know what our class needs to return and can proceed with further considerations.

Considering the pass-through of inherited kwargs

kwargs will become an important topic for this project. Let's take a quick look at a by default generated module inside our dashboard.py file:

        # append an app list module for "Applications"
        self.children.append(modules.AppList(
            _('AppList: Applications'),
            collapsible=True,  # <- coming from DashboardModule-Class
            column=2,
            css_classes=('collapse closed',),
            exclude=('django.contrib.*',),  # <- coming from AppList-Class
        ))

As we can see the module, in this case AppList, gets appended and all the configuration can be done here by passing parameters. Basically inside the grappelli code, there is a class called DashboardModule. This class defines various attributes, like for example collapsible and column. All modules, like AppList, inherit this base class and may add their own attributes like exclude in this example.

For the Graphs-module we are planning to build, we want something similar but...a little bit more complicated. So of course we will inherit DashboardModule like all modules so our module can be easily configured inside the dashboard.py file like all the others. But on top, we might want custom attributes. For example, an attribute that allows us to configure which type of graph should be shown or attributes to set which data is displayed on the graph in the first place. This is similar to what our example AppList provides with it's exclude already. However, we also need more. We will build the graphs with matplotlib (or seaborn; it doesn't matter at this point). These are extensive charting/plotting libraries offering many options. If we would use a bar chart and set everything predefined like bar size, labels, colours, etc. we could only display the same looking charts (which is boring and even worse it may depend on data which is just not suited for this specific visualization). The next issue with this would be that if we want to change the look of graphs later on, we always need to change the code inside our new module. This is not intuitive, doesn't leverage the approach grappelli takes, and will make everything overcomplicated when using our module. So we need a better way. Which in this case is the following strategy. Let's imagine our new custom module will be a simple modified copy of AppList with new attributes, then we could use it like this:

    Graph(
                _('Fancy new graph module!'),
                # Attributes of DashboardModule:
                column=1,
                # Our custom module attributes:
                x_values=['List of x values'],
                y_labels=['List of y values'],
                display='pie',  # type of chart
            ),

Now this would get the job done but would come with all limitations discussed earlier. A more desirable approach (the one we will take) is if we couldn't just give the module when using it the default DashboardModule attributes and our custom Graph-module attributes, but if we could also directly give it any matplotlib (or seaborn) parameters that the plotting functions might accept. Imagine this example using a matplotlib pie chart function which takes the optional argument autopct='%.0f%%' to also display percentages:

    Graph(
                _('Fancy new graph module!'),
                # Dashboard kwargs
                column=1,
                # GraphModule properties
                x_values=[Project.objects.filter(tags__id=tag.id).count() for tag in Tag.objects.all()],
                y_labels=[tag.name for tag in Tag.objects.all()],
                display='pie',
                # kwargs for the called plotting functions
                autopct='%.0f%%',
                random_input="this shouldn't cause an error",
            ),

This would be way more user-friendly and flexible, eliminating all of the earlier concerns. We could have the ease of use of a grappelli module while maintaining most of the power our charting libraries would offer us for a single graph (I am saying "most" because I will ignore advanced charts like multiple combined charts stacked into one plot which will be more problematic as we will see along the way). Also, we can let every sort of configuration for our module live at the same place where it actually belongs.

However, as you may have noticed I put in a useless argument random_input. That's just a little reminder because our considerations of this chapter lead us to two challenges:

  1. We need our class to accept any additional arguments as kwargs to pass them to the plotting functions called
  2. We need to somehow actually pass them along in practice
  3. We need to make sure that a wrong entry in kwargs, for example, if we misspell autopct as autoptc, doesn't break the code (which does actually happen: if we mindlessly pass any additional argument that's not in DashboardModule or our Graphs-module as arguments to our plotting function and there is an argument the plotting function doesn't know it will just break as most plotting functions don't accept kwargs and thus can't handle faulty input)

With that consideration and sort of "rough design planning" we can finally start writing our custom module as we now know what data we want to flow which way and what ultimately needs to be returned in the context to render out the way it should.

Getting our imports ready

Overall, we will need quite a few imports. Let's set them all up right away:

# python imports
import io
import os
import re
from typing import List
from collections import Counter
from datetime import datetime, timedelta

# django imports
from django.utils.translation import gettext_lazy as _  # Note: for django < 4 this should be ugettext_lazy

# library imports
from grappelli.dashboard.modules import DashboardModule
import matplotlib as mpl  # import to set the engine
import matplotlib.pyplot as plt  # import for plotting
import seaborn as sns
import pandas as pd
import numpy as np
import requests

# custom imports
from ..settings._base import GRAPPELLI_CUSTOM_TEMPLATES

The seaborn import isn't really needed as long as you don't want to build seaborn graphs. requests is also just there if you need to fetch data from, let's say Google Analytics or SimpleAnalytics in my case. The last import should be the custom variable in the settings we created earlier. In my case, I am importing from ..settings._base instead of the basic settings.py as I am using a settings file structure as described in an earlier article.

Inheriting a new Graphs-Class on grappelli DashboardModule

It's finally time to start with the code for the actual class.

We will start creating a class, inheriting DashboardModule, and adding some variables. Namely, we assign a default for title and add our template in a corresponding variable (as done in the original grappelli DashboardModule):

class Graphs(DashboardModule):
    title = _('Graph')
    template = os.path.join(GRAPPELLI_CUSTOM_TEMPLATES, 'modules/graph.html')

Next, we will add our own necessary attributes as discussed earlier when we planned for our kwargs:

    x_values = None
    y_labels = None
    display = None  # this will store which type of graph should be shown; e.g. bar or pie
    graph_modifier = {}  # this will be the kwargs we pass down to the plotting function later on

Now we can define a __init__ function for our class. This can be again simply copied from DashboardModule. Here is a version modified for our new attributes and with our first filter to distinct given kwargs:

    def __init__(self, title=None, x_values=None, y_labels=None, display=None, **kwargs):
        kwargs.update({
            'x_values': x_values,
            'y_labels': y_labels,
            'display': display
        })  # give variables to init_with_context function

        # if there is something in kwargs that's NOT there to set DashboardModule or Graphs classes fields,
        # then it's probably there to modify our graphs; hence we keep it in a new variable.
        for key in kwargs:
            if not hasattr(self.__class__, key):
                self.graph_modifier[key] = kwargs[key]  # here we populate our earlier declared graph_modifier dict with kwargs
        super(Graphs, self).__init__(title, **kwargs)  # override title and variables

Next, grappelli modules use a init_with_context function as defined by DashboardModule we will simply overload with our own. Here we call a generate_graph function which doesn't exist yet. However, this part will already finish our function as the only thing we need to do is to define what we will pass as "children" into the template we created earlier and use our graph_modifier attribute to pass on kwargs that was meant for plotting:

    def init_with_context(self, context):

        if self._initialized:
            return

        # passing all kwargs that weren't meant for the DashboardModule or Graphs Class down to generate_graph()
        inherited_kwargs = self.graph_modifier

        # setting the context variable 'children' to a plotting function we will write later that should return an svg as string
        self.children = generate_graph(
            x_values=self.x_values,
            y_labels=self.y_labels,
            display=self.display,
            **inherited_kwargs)

        self._initialized = True

Supporting function: determining valid kwargs

Instead of diving right into the main function to generate graphs, let's go back to our prior consideration about invalid kwargs potentially breaking any plotting function. When using our module in dashboard.py there are various arguments we can pass. Some of them were for the general module setup given by the inherited DashboardModule class, and some by our own new Graph class. We already filtered these in the __init__ function and passed all remaining given attributes that don't belong to either class inside a new dictionary, called it inherited_kwargs, and passed it as kwargs to our (future) plotting function. However, we will need a way to decide if all of these remaining, even though we already know they should belong to our plotting function, are actually valid parameters that the plotting functions actually accept. So we will need a function that filters our inherited kwargs for validity like so:

def filter_valid_kwargs(valid_arguments: List[str], kwargs_dict_to_filter: dict) -> dict:

    valid_kwargs = {}

    for key, value in kwargs_dict_to_filter.items():
        if key in valid_arguments:
            valid_kwargs[key] = kwargs_dict_to_filter[key]

    return valid_kwargs

What this function will do for us is basically accept a list of arguments we know are valid, and checks if there are corresponding keys in a given dictionary (with the kwargs we will put in actually are). If there is a match it will be added to our resulting dictionary, if not it will get dropped.

NOTE: this doesn't ensure full fail safety! We don't check if the values of the keys actually are of the right type, like integer or string. A more sufficient function would take a dictionary consisting of allowed arguments as keys and their allowed types as values to verify given kwargs. However, to build the lists of valid arguments later on we have to copy them from the documentation sites of matplotlib and seaborn and....they are...a lot. I was simply too lazy at this point since I only do this project to play around with grappelli for myself. But if you write your code by this article for production use...keep that in mind and plan for a few extra minutes to write proper validation dictionaries.

Supporting function: cleaning SVG strings for the template

Now would be a great moment to finally start writing the main graphing function. But we did the most supporting work already, so let's finish it up since we are already at it.

We will return an SVG string from matplotlib later on. Spoiler: the code we get will be messy with metadata and stuff we don't need and that will disturb our injection of the string into our HTML template. So we need a function to extract only the <svg ... </svg> part of the code. Another problem: the SVG will have fixed width and height attributes which will be annoying. Easy resizing isn't possible this way. The most straightforward solution is to just remove them. This way the SVG fill the entire HTML container of our template and we only have to worry about the size of our actual module on the dashboard without constantly adjusting the SVG. Overall the beauty of vector graphics is that they can scale easily, so we should preserve this behaviour and just get rid of the fixed dimensions.

We will achieve both of these goals with the following simple function:

def clean_svg_string(svg_string) -> str:
    extracted_svg = re.search(r"(<svg)([\s\S]*)(<\/svg>)", svg_string.getvalue())[0]
    extracted_svg = re.sub(r"(width=)([\s\S]*)(height=\")(.*)(?=viewBox)", '', extracted_svg)
    return extracted_svg

The first line of the function extracts only the <svg ... </svg> range of the code we actually need. About the second line...let's explore it a little bit more in detail.

After the first line our extracted_svg variable string prints out like this:

<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="432pt" height="432pt" viewBox="0 0 432 432" xmlns="http://www.w3.org/2000/svg" version="1.1">
 <metadata>
  <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
   <cc:Work>
    <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
    <dc:date>2022-07-11T19:02:19.570525</dc:date>
...

Notice how there are line breaks. This is the reason why inside the regex we use [\s\S]* (any non-whitespace or whitespace character any time) instead of just .* (any character any time). But besides that, here we need a way to extract only width="432pt" height="432pt" and get rid of this specific part.

In the regex of the second line what we basically do is look or width= as our starting point, some characters, height=", some characters, and then we set as part of our regex pattern that it should end (but not include!) with viewBox. Or to put it in a more visual way using regex101 (which is a site I am totally in love with by the way):

Regex preview on regex101 to show how the regex pattern finds width and height inside the svg string

"Isn't this overengineered?" you might ask. Yes, it is. However, let's consider the implied working hypothesis used so far. All of this does only work reliably because of one reason: we can take assumptions we are sure to be true. For example, we expect there only to be one opening SVG tag and one closing SVG tag. As that's what we generate in our function later, we can be quite sure about that one. However, are we sure the words width and height will never ever occur in the string no matter what we do to the plot? Maybe. But I am not 100% convinced and don't want it to break by being afraid of a bit more complex regex than maybe actually needed. That's why this pattern looks overly complicated. For the first thesis, I am not sure, but I am pretty confident that width and height won't appear in this pattern ever again. Another solution would be to search the pattern and only substitute the first match to be even on the safer side but...that would be even too overly cautious even by my standards.

Anyway. So far we have a function that gets us the SVG tag and all it encloses from whatever matplotlib puts out and removes fixed size properties for display. Neat.

Core function: actually generating graphs

Annnnd the moment we were all waiting for: creating the main plotting function. Or maybe not. Probably you got the idea about my plan with "inherited kwargs" by now, only wanted to copy the quick code for the class, and already did write a 2-liner plotting your chart into an SVG string. In this case..thanks for reading. If you want to hang around anyway or are not too much into matplotlib already, here is the highlight chapter coming for you ;)

What graphs will we build and how will they work?

  • Basic pie chart from given lists of x and y values
  • Basic bar plot from given lists of x and y values
  • Stacked bar plot (this will be more complex fun) from a given dataframe
  • A static...bubble...line...chart..thingy?! Fetching data from SimpleAnalytics to give us some website statistics :)

Let's start with a quick refresher on what we actually pass to our function in the prior written init_with_context function:

        self.children = generate_graph(
            x_values=self.x_values,
            y_labels=self.y_labels,
            display=self.display,
            **inherited_kwargs)

So that will be the foundation for our new function with a few basic definitions:

def generate_graph(y_labels: List[str],
                   x_values: List[int],
                   display: str = 'pie',
                   colors: str = None,
                   **kwargs) -> str:

    # create a string container for finished svg string
    plotted_svg = io.StringIO()

    # set an engine for matplotlib
    mpl.use('Agg')

    # turn interactive plotting off; if we try to plot interactive to the console our server will crash!
    plt.ioff()

    # create a new figure to plot into
    fig = plt.figure()

Next I'll add an optional line. I just like seaborn colour pallets. If you don't plan on using seaborn, just skip the following line:

    # set colors if not specified otherwise
    if not colors: colors = sns.color_palette('pastel')[0:len(x_values)]

The number of colours gets limited to the number of data points. However, it seems to still work fine without [0:len(x_values)].

We plan on using several charts.

NOTE 1: I will use if...else statements here. This isn't a good idea as python 3.10 offers us a better way by using switch...case statements. However, to keep it compatible for now I will stick to if...else.

NOTE 2: Even if you would use switch...case statements, this still wouldn't be optimal as the function gets long pretty quickly when adding even just a few graphs. Having a dedicated function for each graph that just gets called would be better. I keep it quick and simple with if...else for now.

The pie chart

Let's start with the easiest chart of this project, the pie chart. It just takes 3 lines of code (you could even write it in just one line even though it would hurt readability) so I will let it just write out here with its comments as the only explanation:

    if display == 'pie':

        # The inherhited kwargs could be anything a developer types
        # If a keyword is typed in that's not understood by the according matplotlib function we get an error
        # Hence we can't just pass any keyword from our interited kwargs down into our plotting function.
        # So we retrieve a list of all modifiers available to our plot type:
        # (Source of arguments: https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.pie.html)
        valid_arguments = ['explode', 'autopct', 'pctdistance', 'shadow', 'labeldistance',
                           'startangle', 'radius', 'counterclock', 'wedgeprops', 'textprops',
                           'center', 'frame', 'rotatelabels', 'normalize', 'data']
        # arguments we already pass as "required", x, labels, and colors are removed already as they can't be double (and are fixed part of the Graph class)

        # Now filtering the interhited kwargs for valid arguments to the plotting function:
        valid_kwargs = filter_valid_kwargs(valid_arguments, kwargs)

        # And..just plotting it out to the initially created figure
        plt.pie(x=x_values, labels=y_labels, colors=colors, **valid_kwargs)

Now we can already finish the generate_graph function:

    # Where is the "else:" you might ask
    # Note how this isn't a good solution, but nothing will crash if "display" isn't set to "pie"
    # In this case the function will simply display an empty svg as we have no "else" statement; blank white output.

    # save the figure in our svg string as...well, "svg string" ;)
    plt.savefig(plotted_svg, format="svg")

    # close the figure so it doesn't get displayed and break (we don't run this in a jupyter notebook or so...)
    plt.close(fig)

    # return the plotted string to our prior created cleaning function and...that's it!
    return clean_svg_string(svg_string=plotted_svg)

Now we already have a working module. If you go back to the part where we "imagined" how we would want a Graph module in our dashboard.py well...it now works that way! So you can just copy this "imagined" code as it became now a reality and enjoy your graph :)

You can however still further read the article to see how I layout my dashboards, what data might be interesting (what I wanted to display and considered useful), and how to implement further graphs.

The simple bar chart

Alright, not based on the pie chart you might guess how utterly complex the bar chart will turn out. We will need 3 entire lines of code...so..uhm...well. That's it:

# reminder: we prior had an "if display == 'pie'" and now we can just extend on this:

elif display == 'bar':
    # drawing the bar chart
    plt.bar(x=[x for x in range(0, len(y_labels))], height=x_values, **valid_kwargs)
    # adding tick labels
    plt.xticks(ticks=[x for x in range(0, len(y_labels))], labels=y_labels)
    # and I know I will always have text labels; hence I set the text orientation to 45° to make some space
    fig.autofmt_xdate(rotation=45)
    # NOTE: if you wanted this to be more general purpose you could define an argument for this like we did with "colours"
    # on the Graph class level and wrap this last line in a simple if statement like "if bool_rotate_text: ..."

That's all for this one. When we extend it for the stacked bar chart in the next chapter, however, things get more interesting.

The stacked bar chart

(following shortly)

Adding SimpleAnalytics website statistics

(following shortly)

Usage

(following shortly)