What is a Dependency?

In this document, dependencies are import relationships between Python source files. If file a.py imports some code from file b.py, we say that a.py depends on b.py, or that a.py has a dependency on b.py.

Controlling the dependencies between the various parts of your code (or between projects that your code uses) is a powerful way to increase the reusability of your code: the least amount of dependencies a body of source code has, the greater the likelihood that this code will be usable in the future. Dependencies cause codes to break due to the independent evolution of modules and changes in the interfaces. Even within a single project, controlling dependencies between layers is the essence of modularity.

Snakefood allows you to automatically generate a visual graph of those dependencies. When you produce these kinds of graphs, you will often be surprised at how certain parts of your code inadvertently become tied together. Dependency graphs allow you to view the relationships clearly and will generate questions about the high-level organization of your code and allow to improve modularity and reusability.

How Packages Are Automatically Found

For each file or directory that you specify on the command-line, sfood automatically figures out which root it lives in. What we call a “root” in this document is a directory that should be in PYTHONPATH if you were to import the given files from a script.

Snakefood finds the package roots,

1. by walking up the directory hierarchy of each file until it finds a directory without a __init__.py file, or if no package root is found this way,
2. by walking down the directory hierarchy until it finds such roots.

All of the package roots found this way are automatically prepended to the Python import path, so you do not have to configure your PYTHONPATH before you invoke sfood. sys.path is in effect when processing imports. It is simply augmented to include the roots that snakefood found

The roots are added to the module path and used to separate the filename in two parts: the package root, and the filename relative to that root (see the section on dependencies format below).

Following Dependencies

By default, sfood does not follow dependencies, that is, if you ask it to process a.py and it finds that it has a dependency on b.py, the file b.py does not get processed for its dependencies.

Use the --follow option to tell sfood to follow its dependencies. Files for each target dependency that were found are automatically examined for dependencies, and the process continues until all leaves are found (we check for cycles too).

Restricting Dependencies

Normally you will want to find out the relationships only between files in the packages that you provide on the command-line. For example, you will probably not be interested to find out about dependencies to the modules in the standard library that comes with Python.

You can filter out the dependencies by yourself using the usual UNIX tools, like grep or sed. But since this is a typical case, there is a convenient --internal option to sfood that automatically limits dependencies to only those files living in the package roots corresponding to the set files that you specify on the command-line.

You can restrict the dependencies even further, to only the list of files that have been processed, that is, the list you specified on the command-line in the first place. You enable this by using the --internal option twice. This is a convenient feature when you want to chart the dependencies of only the files in a subdirectory of a larger project.

Ignoring Unused Imports

With the --ignore-unused option, sfood will automatically ignore dependencies motiviated by symbols imported but not used. This determination is performed using the same conservative and safe heuristic as is used in sfood-checker.

The resulting list of dependencies will always be smaller when running with this option than without. Note that the resulting dependency tree is the same that you would obtain should you clean up all the analyzed code using sfood-checker, to hunt for unused imports.

The Format of Dependencies Files

The format of dependencies is really simple:

((<source_package_root>, <source_file.py>), (<dest_package_root>, <dest_file.py>))

where the package_root files are the directory names at the root where the module files are found, and the file.py names are the Python filenames relative to the corresponding root.

Each line is a valid Python tuple expression, so you can easily write a Python script to process them using a line like:

for dep in map(eval, sys.stdin):
    ...

and output them like:

dep = (froot, fn), (troot, tn)
print repr(dep)

Caching Raw Dependencies

The process of building a nicely filtered dependency graph is iterative, you will typically massage the dependencies to highlight the relationships that you care about. Since calculating the dependencies can be a slow process (and filtering and graph generation is not), we recommend to save the output of sfood to a file and work from that.

Warnings

You may see a lot of warnings when you run sfood. This is normal. There are a few reasons for this:

• The code you are analyzing requires some external packages that are not installed or not in your PYTHONPATH;

• The from-import Python syntax is ambiguous; for example, in the following code, it is not clear whether table is a module or an object defined in the database module:

  from database import table
  

  Therefore, sfood does not normally print out warnings for these. If you want to see a list of those failed imports, run it with the --verbose option.

In eitehr case sfood keeps running and produces all the other dependencies that it finds.

Nodes Without Dependencies

To insure that all the file nodes show up in the graph, for each processed file we output a line like this:

((<source_package_root>, <source_file.py>), (None, None))

The graphing tool interprets that to mean it has to at least create a node for <source_file>.py, even if it has no dependency. Scripts you write to filter the dependencies need to be able to interpret those lines appropriately.

Pragmas: Marking Dependencies as Ignored

Sometimes when an import statement is wrapped in a conditional, you may want to avoid generating a dependency for that statement, for example:

try:
    import superhero
except ImportError:
    # superhero not available. We fallback on evilgenius.
    import evilgenius

In this example, you may want to avoid having snakefood follow dependencies to superhero, because your software can run fine without it (that is, with evilgenius). To that effect, snakefood looks for a string after the import statement, and uses that string as a hint to modify its actions, for example, this will tell sfood to not output the dependency for superhero:

try:
    import superhero; 'OPTIONAL'
...

This is especially useful with the --follow option, when you want to avoid dragging a large dependency in the dependency list.

Using Standard UNIX Tools

Since dependencies are simple line-based Python expressions, you can use grep or sed to filter out or modify unwanted lines:

cat raw.deps | grep -v /usr/lib/python | sfood-graph > out.dot

There is no formula for filtering or reformatting the dependencies; it depends on your codebase and what you want the graph to show.

Using the Clustering Tool

A useful operation is to transform the relative filenames into logical groups and to remove redundant lines. We call this “clustering”. A common example is to lump together all the filenames that start with a particular directory prefix.

You could do this with sed but you also need to remove redundant lines to do it properly, i.e. after simplification of the relative filenames, multiple lines will be equivalent. sfood-cluster does that automatically. You create a file of cluster names:

pack1
pack2

and a dependency file will be transformed from this:

(('/myproject', 'pack1/person.py'), ('/myproject', 'pack1/employee.py'))
(('/myproject', 'pack1/person.py'), ('/myproject', 'pack1/manager.py'))
(('/myproject', 'pack1/person.py'), ('/myproject', 'pack2/boss.py'))

to this:

(('/myproject', 'pack1'), ('/myproject', 'pack1'))
(('/myproject', 'pack1'), ('/myproject', 'pack2'))

Here is how to use the sfood-cluster tool:

sfood /myproject | sfood-cluster -f clusters | sfood-graph > myproject.dot

Pipeline for dependency graph with clustering.

You can either create the clusters file manually, or with a find or ls command in your source tree."

Using a Pragma to Ignore an Unused Dependency

If you are importing a file for its side-effects only, sfood-checker has no way to detect this and will report the import as unused. To avoid this, use a pragma after the import statement to disable the warning:

# Importing for side-effects only.
import injectrace; 'SIDE-EFFECTS'

Enforcing Dependency Relationships on Commit

If a codebase has some package dependency relationships that should be enforced, for example, that anything in package root.core should not depend on anything in package root.util, you can easily write a post-commit hook in your favorite source code versioning system that will run sfood on an up-to-date checkout and grep for the offending relationships. Such a script could send an email to the checkins list, or even refuse to commit if the offending dependency occurs.

Splitting a Codebase

It is often required to extract a portion of a codebase outside a source tree, in a way that the extracted software will still be functional. In other words, you sometimes have to extract a script and all the dependencies that it drags along with it to remain usable.

You can easily use snakefood for this purpose: run sfood --follow --internal on the given script, and it should produce the list of dependencies that it needs to continue functioning. Flatten these dependencies into a list of filenames with sfood-flatten and copy the files somewhere else.