OpenSolaris DTrace for Home Media Servers, Revisited

A few weeks ago, we discussed using DTrace for automatically updating media servers when you upload new content.

Yesterday though, I discovered that my D script didn’t work any more. I uploaded new songs to my home server, and expected the music daemon to re-scan the music directory, but nothing happened.

That teached me an important lesson about DTrace, and here’s what I learned:

The Initial Idea: Automatic Media Scan on Update

As a reminder, this is the idea of my DTrace script:

We tap into the syscall::open*: probes and listen for calls to them where O_WRONLY or O_RDWR is set.
Then we check if the path to the file being opened for writes is inside our media directory.
If so, we send a SIGHUP to the media server process (in my case it’s mt-daapd and mediatomb) which will trigger them to re-scan their media directory so they notice when there’s new content to serve.

The effect is simple: Upload new music, and the music server immediately knows there’s new stuff to play, even though it doesn’t support inotify or other forms of file change detection.

The DTrace Dynamic Variable Drop Error Message

But somehow, this didn’t work yesterday for me. Instead, during the rsync that loaded new music to my home server, the logs showed:

dtrace: 4 dynamic variable drops
dtrace: 18 dynamic variable drops
dtrace: 39 dynamic variable drops
dtrace: 6 dynamic variable drops
dtrace: 10 dynamic variable drops
dtrace: 2 dynamic variable drops
dtrace: 1 dynamic variable drop

What’s a “dynamic variable drop”?

DTrace stores dynamic variables in kernel memory. Kernel memory is precious and limited and you don’t want to run out of it.

So in order to protect the system from running out of kernel memory because too many of it has been allocated for DTrace dynamic variables, it will simply stop allocating more memory for dynamic variables after reaching a certain limit and issue a warning message like we see above.

Dynamic variable drops occur when you use too many variables without freeing them again (by assigning them to be zero).

More about dynamic variable drops can be found in Bryan Cantrill’s DTrace tips slide deck (no link, sun.com no longer exists).

In other words: My D script had a memory leak.

But where?

Finding the Memory Leak

After spending some more time with my DTrace code, it dawned on me:

First, we take a note of the file name inside a DTrace thread-local variable called self->file whenever we enter the open function when it’s called with the right flags:

syscall::open*:entry    /* Take note of the arg0 pointer for later use. */
/ !flag && (arg1 & (O_WRONLY | O_RDWR)) /
{
  self->file = arg0
}

Then, we do the real work when returning from the function call, but only when certain criteria are met, and reset our self->file variable to zero when we’re done:

syscall::open*:return
/
  !flag &&       /* Exit if we've already found a hit. */
  self->file &&  /* Exit if we didn't take an arg0 note. */
  (              /* More fancy directory testing. */
    (copyinstr(self->file, dirlen) == dir) ||
    (
      copyinstr(self->file, 1) != "/" && (
        substr(
          strjoin(strjoin(cwd,"/"),copyinstr(self->file)), 0, dirlen
        ) == dir
      )
    )
  )
/
{
  flag = 1;
  self->file = 0;
}

But what happens when filter criteria are not met at the time our function call returns?

Sure, we’re not interested in these cases, but this also means we’ll leave our self->file variable hanging in the air!

And since it’s thread-local, a new one will be generated for the next thread. And another one for the next thread after that. Over and over again, until we run out of DTrace dynamic variable space.

Not pretty.

The Solution

So I needed a solution that would make sure all instances of self->file are properly zeroed when not needed, or better yet: Get rid of the variable altogether.

Salvation came as I discovered, that DTrace can give me the whole fileinfo_t structure for a given file descriptor (thanks, Robert!), which is available as arg0 from the syscall::open*:return probes.

Since fileinfo_t contains both the flags that the file was opened with and also the full path to the file (including any relative or absolute prefixes), there’s no need to use syscall::open*:entry at all and we can do all our work in syscall::open*:return while making the script much more simple:

#!/usr/sbin/dtrace -wqs

/*
 * dirtrap.d
 *
 * Detect when new files have been written to a given directory.
 * Start a command when this happens.
 */

BEGIN      /* Initialize stuff. */
{
  dir = $1;
  dirlen = strlen(dir);
  flag = 0;
}

syscall::open:return,
syscall::open64:return
/
  !flag &&          /* Exit if we've already found a hit. */
  fds[arg0].fi_oflags & (O_WRONLY | O_RDWR) && /* Check if opened for writes */
  substr(fds[arg0].fi_pathname, 0, dirlen) == dir /* Check directory */
/
{
  flag = 1;
}

tick-60s
/flag/    /* Periodic check if we need to do something. */
{
  system($2);
  flag = 0;
}

Note that we also got rid of the complicated path-wrangling code, as fi_pathname will always have the full pathname, no matter whether we opened a file relative to the current working directory or an absolute path. Neat!

We also got rid of all thread-local variables, which means there’s no way we can create more than the three variables mentioned in the script. Bye, bye “dynamic variable drop” errors!

Lessons Learned

Always check all possible decision paths in your DTrace scripts, even those that you aren’t interested in.
Only use thread-local variables when really necessary, and then make sure you properly release all of them back to DTrace.
There’s always a better way to do stuff, you just need to look hard enough :).

Make-up and Cliffhanger

To make up for the ~~crappy~~ slightly less optimal code, I’ll write my next post about how to nicely wrap this DTrace script inside a Solaris SMF service, so it can be easily managed on your home server. Then we’ll wrap both the DTrace script and the SMF service inside a shell script, that will make installing and activating the whole thing easy to administer.

Your Turn

What are your favourite DTrace “duh” moments? What did you learn from them? What are your favourite DTrace hacks?

Share them in the comments section and start hacking DTrace!

This post is obsolete