Evaluation Utilities¶

There are several make commands and utility scripts in the repository's utilities directory. They help inspect, test, measure and report of performance of pymwp. These tools are not shipped with the distributed version of pymwp, but are available from the source code repository.

AST Generator
Convert C files to ASTs using pycparser.
Machine Details
Captures host machine details at runtime.
Plot Results
Make table plots of pymwp results.
Profiling
Inspect functions runtime execution details.

Required dependencies

First install required dependencies for running utilities:

python -m pip install -r requirements-dev.txt

AST Generator¶

This is a utility script that reads and parses a C file, or a directory of C files, then generates an AST. It uses gcc and pycparser, then writes the AST to a file. This script is mainly useful for generating/updating test cases for unit testing, and for inspecting AST structure and nodes. The parser options are hard-coded, and assumes C files have no custom headers, and that gcc is an existing system C compiler.

Usage:

python3 utilities/ast_util.py IN_PATH OUT_DIR

Parameters:

Name	Type	Description	Default
`IN_PATH`	`str`	A C file or path to a directory of C files.	required
`OUT_DIR`	`str`	Directory where to save AST.	required

Machine Details¶

Capture details of the current machine and runtime.

Usage:

python3 utilities/runtime.py OUTPUT_DIR

Parameters:

Name	Type	Description	Default
`OUTPUT_DIR`	`str`	Directory where to write the output.	required

Plot Results¶

Makes a table plot of analysis results.

Usage:

python3 utilities/plot.py  --in IN --out OUT --fmt FORMAT

Parameters:

Name	Type	Description	Default
`--in`	`str`	Directory of analysis results.	required
`--out`	`str`	Directory where to save generated plot.	required
`--fmt`	`str`	Plot format: `tex` or `plain`.	required

Profiling¶

Profiling reveals how many times different functions are called during analysis. Profiling is based on Python cProfile.

See: https://docs.python.org/3/library/profile.html

Single file profile¶

Single-file profiling works on both pymwp installed from package registry or when running pymwp from source, because it requires only the standard Python module cProfile.

Usage:

Release versionRunning from source

python -m cProfile pymwp --silent -s ncalls INPUT_FILE

python -m cProfile -m pymwp --silent -s ncalls INPUT_FILE

Parameters:

Name	Type	Description	Default
`INPUT_FILE`	`str`	Path to input C file.	required
`--silent`		Mute pymwp analysis output.	required
`-s`	`str`	Specifies cProfile output sort order.	required

See: https://docs.python.org/3/library/profile.html#pstats.Stats.sort_stats

Additional arguments of cProfile or pymwp can be added similarly.

Multi-file profile¶

Profiler utility module is a wrapper for cProfile. It enables profiling directories of C files. The results of each execution are stored in corresponding output files.

One outputs is displayed for each profiled file:: done-ok profiling subprocess terminated without error, note: even if pymwp analysis ends with non-0 exit code, it falls into this category if it does not crash the subprocess.; error profiling subprocess terminated in error.; timeout profiling subprocess did not terminate within time limit and was forced to quit.

Usage:

Profile all repository examples:

make profile

Run with custom arguments:

python utilities/profiler.py --in IN --out OUT --sort SORT --timeout SEC
       --lines LINES --skip SKIP --only ONLY --extern --callers --save --help

Parameters:

Name	Type	Description	Default
`--in`	`str`	Directory path to C-files [default: c_files].	required
`--out`	`str`	Directory path for storing results [default: `output/profile`].	required
`--sort`	`str`	Property to sort by [default: `calls`].	required
`--timeout`	`int`	Max. timeout in seconds for one execution [default: 10].	required
`--lines`	`int`	Number lines of profiler stats to collect e.g. to profile top 10 methods, use `--lines 10`.	required
`--skip`	`str`	Space-separated list of files to exclude e.g., `--skip dense infinite_2` will not profile matching files.	required
`--only`	`str`	Space-separated list of files to include e.g., `--only dense empty` will profile only matching files.	required
`--extern`		Exclude package external methods from cProfile results	required
`--callers`		Include function caller statistics.	required
`--save`		Save pymwp analysis results [default: False].	required