Hello everyone. I like Jokosher. I think its a cool piece of software, with the right direction and a great bunch of people working on. Sure, I am biased, but nonetheless, Jokosher is an exciting project that has the opportunity to really redefine how we record music on a free software platform. But we have a problem…

…we need more testers. We need more people testing Jokosher, finding problems and reporting those problems. Now, some people are doing this and reporting bugs and reporting things on our [excellent forums](https://www.jokosher.org/forums/), but what would be *really* useful would be if some people could have a go at debugging Jokosher, and this in turns means being able to debug a GStreamer application.

We had a meeting about the new Jokosher QA Team yesterday and I agreed to write up a quick guide to getting started doing this kind of debugging. Hopefully this will help some of you get started in this incredibly important aspect of Jokosher. If you read this guide, it should give you a good solid grounding in how to debug Jokosher and contribute essential debugging information to our developers who can fix the problems. 🙂

## A bit of preparation

Before you begin, I recommend you get a quick overview of how GStreamer works. You basically need to know that you string together elements (plugins that do something specific such as reading in a file, decoding music or applying effects) and those elements are linked and have their pads connected together. I wrote a few little guides that explain how GStreamer and Gnonlin work. Although these guides cover how to write code, read them and understand the theory in them and it should give you a firm grounding in GStreamer. They are at:

* [Getting started with GStreamer with Python](https://archivedblog.jonobacon.com/?p=750)
* [Using Gnonlin with GStreamer and Python](https://archivedblog.jonobacon.com/?p=851)

You should also take a look at the diagram of the Jokosher pipeline:

* [Jokosher GStreamer pipeline](https://jokosher.python-hosting.com/attachment/wiki/CodingTeam/pipelinemap.png?format=raw)

This pipeline shows all of the elements and how they fit together in Jokosher.

In terms of software, it is recommended you are running Jokosher from SVN trunk and running a CVS copy of GStreamer. Instructions for this can be found [here](https://jokosher.python-hosting.com/wiki/InstallingCvsGstreamer).

## Getting started

The first thing you should do is initialise your CVS GStreamer setup. If you are using the guide linked above, you run a script that initialised CVS GStreamer for you. For me I run:

jono@forge:~$ bin/gst-head

The next step is to find a bug to work on. Take a look at our [bug tracker](https://bugs.launchpad.net/jokosher/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=Unconfirmed&field.status%3Alist=Confirmed&field.status%3Alist=In+Progress&field.status%3Alist=Needs+Info&field.status%3Alist=Fix+Committed&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package=) and find a bug to do with playback or recording. This bug should preferably include some steps to reproduce the bug.

The next step is to run Jokosher in debug mode, reproduce a bug and then explore the log file. To do this, move to the directory where the main Jokosher code lives (namely where the `JokosherApp.py` file is present) and run the following command:

jono@forge:~/jokosher/trunk/Jokosher$ GST_DEBUG=3,python:5,gnl*:5 ./Jokosher -g > log 2>&1

This command does a number of things. Lets break it down:

* `GST_DEBUG=3,` – set the GStreamer debug detail level to 3 (1 is least detailed, 5 is insane uber-detail).
* `python:5,` – set the python debug level to 5.
* `gnl*:5` – set the Gnonlin debug level to 5.
* `./Jokosher -g` – run Jokosher and pass it the `-g` option which directs all debug output to the log file.
* `> log 2>&1` – send all debug output to a file called `log`.

Run the command and reproduce the bug by performing actions in Jokosher until the bug occurs. Try to remember the steps you follow – you will need to match your actions to different parts of the log file. If you get a GStreamer error message (one of the errors that says *Arggh!* in Jokosher) that will also be logged to the log file.

## Looking at the log file

When you run the last command you set GStreamer debugging into its default mode, which rather handily adds ANSI colours to the log file to distinguish between different types of information. To understand these colours, you need to run less to look at the log file. Run this command:

jono@forge:~/jokosher/trunk/Jokosher$ less -R log

It will then say `”log” may be a binary file. See it anyway?`. Type `y` to continue. The log file will appear and you will be presented with thousands of lines of debug code. Don’t be put off though, you will be skipping through large chunks of this code.

The debug log consists of a bunch of different types of information. A typical line looks like this:

0:00:06.855443000 15857 0x8249040 INFO GST_ELEMENT_FACTORY gstelementfactory.c:377:gst_element_factory_create:
creating element “bin” named “Volume_fades_bin”

If you look at the above line, there are two key bits of information you need to be aware of. The `INFO` bit tells you the type of debug message this line is and the content to the right of it is the message itself. There are a few different types of message:

* `INFO` – general information about what GStreamer is doing.
* `DEBUG` – debugging information from the application itself. More on this below.
* `LOG` – more logging information, typically used by Gnonlin.
* `WARN` – GStreamer warnings.

…and I believe there is an `ERROR` type as well, which are GStreamer errors. I seem to remember seeing this, but I am not 100% sure.

To the right of the type of message is the actual log message, which in the above example is `GST_ELEMENT_FACTORY gstelementfactory.c:377:gst_element_factory_create: creating element “bin” named “Volume_fades_bin”`. In this message a *bin* called `Volume_fades_bin` is being created.

You job now is to read through the lines in the debug log, matching the different parts to the elements and connections in the pipeline map, and try to figure out where the bug is occurring. Reading the log will be rather confusing at first, but after a few times you will get the hang of it, and after asking some questions about what debug messages mean, you will get the hang of how to understand the log file and see what is going on.

The log file is chronologically organised – the content is appended to the file as Jokosher runs. When Jokosher initialises, the pipeline is constructed, and as different things happen, the pipeline is updated and adjusted as Jokosher runs. So, as an example, when you add an instrument, a sub-pipeline (a small bunch of elements that are part of the wider pipeline) is added to the main pipeline. When this happens, some elements from the main pipeline need to be un-linked, the new sub-pipeline created and then hooked into the main pipeline. You can see how this works by reading the log file and tracking how the pipeline map is created.

Aside from GStreamer debug messages, there are also a number of debug messages written into Jokosher by our coders. The `-g` switch that you pass to Jokosher switches on these debug messages so they are saved in the log file. If you search the log file for `python` you can see these messages. You can then search the Jokosher code for `Globals.debug` to see where the debug messages are added. So, as an example, imagine this line in a Jokosher source code file:

Globals.debug(“foo bar”)

This line would generate the following debug log message (as an example):

0:00:06.896163000 15857 0x8249040 DEBUG python Globals.py:147:debug: foo bar

So you can match the contents of the log file to different bits of code.

Navigating the log file with the `less` command uses a few keyboard commands which you can learn by running `man less` and reading the information or by [reading about it here](https://linux.about.com/library/cmd/blcmdl1_less.htm). The key skill is typing a slash and a search term will highlight the search term throughout the file (e.g. type `/myterm` will highlight all instances of `myterm` in the log file).

Debugging is like forensics in CSI – you piece together a picture of the crime from the evidence you have, the main evidence being the pipeline map (which shows the GStreamer structure inside Jokosher), the debug log and the actions in Jokosher that were performed to trigger the bug. You use this evidence to narrow in on the problem. It is a fun, challenging and rewarding process when you explore the evidence and then discover what the bug is. They key thing then is sharing your findings with the rest of the team so that the bug can be fixed.

You should share any findings on the bug report of the bug you are investigating. If a bug report does not exist yet, create one and add your findings.

## Getting help

Debugging is a science that cannot be learned in a single evening, and you will need to ask plenty of questions as you learn to debug. Do hang in there though! Debugging is an *incredibly* useful thing for you to do to help the Jokosher project. You are welcome to ask questions in `#jokosher` on Freenode, post to our [mailing list](https://mail.gnome.org/mailman/listinfo/jokosher-devel-list) or ask in the [Jokosher forums](https://www.jokosher.org/forums/).

Good luck, and thanks for getting involved in this fun, exciting and important project. Feel free to ask questions or clarify points in the comments on this blog entry! 🙂