This page documents Report Generator, an obsolete collection of Perl scripts that Polygraph versions prior to 2.8.0 were using to build reports. The information on this page is specific to versions prior to 2.8.0, and documented functionality is likely to be removed starting with version 2.9.0. Current reporting tools are documented elsewhere.
1. For the impatient
2. Introduction
3. Before you can use Report Generator
3.1 Perl
3.2 Gnuplot
4. Configuring Report Generator
5. How to generate reports
6. Caution
# run the test, possibly on several clients and servers clt1> polyclt --log clt1.log ... clt2> polyclt --log clt2.log ... ... # copy logs from each client (and server) to a local directory mon> scp clt1:clt1.log clt2:clt2.log ... test123/logs/ # label the logs and generate a report mon> label_results test123 test123/logs/clt*.log mon> make_report test123 # view file:/tmp/polyrep/test123/index.html
Web Polygraph includes a set of tools for automated report generation. These tools, collectively called Report Generator, use binary logs and other sources of information to generate reports that can be then viewed with your favorite browser. The reports include ``executive summary'' section for the boss, and more detailed analysis for the rest of us. Some common performance and configuration problems can also be automatically detected.
Report Generator uses two external packages that you must have installed to use the Generator. These two packages are briefly described below. If you do not have them on your computer, follow the links to download and install the software.
3.1 Perl
Perl is a must-have package for any professional Unix user. It is probably installed on your box already. Make sure you have Perl version 5.003 or later.
example> perl -v This is perl, version 5.004 example> which perl /usr/bin/perlPolygraph assumes that your Perl interpreter resides in /usr/bin/. If that is not the case, you will need to adjust the first line of all Perl scripts in Report Generator. (Eventually, ./configure will do that for you). For example, if your Perl interpreter is called /usr/bin/perl, then first lines of all Perl scripts should look like
#!/usr/bin/perl -wIf the first lines contain incorrect interpreter file name, you will get funny errors like this:
example> ls -l test.pl -rwxr--r-- 1 alan smithee 44 Dec 27 09:17 test.pl example> ./test.pl ./test.pl: Command not found. example> perl test.pl Well, hi there!If you do not want to modify scripts, you can run script X as ``perl X'' or create a /usr/bin/perl link pointing to the real Perl interpreter you are using.
3.2 Gnuplot
Gnuplot is one of the best plotting (graph generation) tools for Unix. We recommend version 3.6 or later.
example> gnuplot gnuplot> show version Unix version 3.7 patchlevel 0 last modified Thu Jan 14 19:34:53 BST 1999 http://www.uni-karlsruhe.de/~ig25/gnuplot-faq/When installing gnuplot, you may want to enable the PNG ``terminal driver''. Report Generator uses PNG image format because recent gnuplot distributions do not support GIF due to some stupid patents. If you want to use image format other than PNG, try changing the Generator configuration file (see plotter_terminal occurrences in reportgen.cfg file).
All Report Generator tools are located in polygraph/tools/ReportGen directory of the distribution. There are several Perl scripts that use the same configuration file called reportgen.cfg. You may want to edit that file to tune Report Generator for your local environment.
At the time of writing, it is better to use Report Generator from the original source directory. Calling scripts ``remotely'' will not work well. The default configuration file is tuned appropriately.
Report generation takes 2 simple steps described below.
Step 1: Label your results
The label_results script ``names'' a group of logs. Or, to be precise, the script assigns a label to information extracted from one or more log files. That label can be used later to generate reports and plot graphs.
Let's assume that I have run a MyMix-2 no-proxy test on three client and three server machines called nlx01 - nlx06. MyMix-2 is very similar to PolyMix-2 except for phase durations. Assume the binary logs were collected in the /tmp/logs/ directory on each machine. I want to build a report on the machine called stan that happened to have Report Generator installed.
First, I need to copy client logs to stan:
You can use rcp or whatever you normally use for copying the files between two machines. For simplicity, I assume that /tmp/logs/ directories contain just the files I need. If they contain other files as well, my file masks would have to be more elaborate.stan> scp -p nlx01:/tmp/logs/*log \ nlx02:/tmp/logs/*log nlx03:/tmp/logs/*log \ /tmp/logs/Now I want to ``label'' the logs I've just copied. Suppose I want to name them test1. The easiest way to do it would be
stan> cd /usr/local/polygraph/ReportGen/ stan> ./label_results test1 /tmp/logs/*log ./label_results: creating: `/tmp/polyrep/test1/clt.All.lx' ./label_results: creating: `/tmp/polyrep/test1/clt.Traces.lx'Label_results extracted aggregate statistics (using lx), placed it into the clt.All.lx file, and then extracted some trace data (using ltrace) into clt.Traces.lx. You can view .lx files as they are in plain ASCII format, but you do not not care about that unless something goes wrong.
While the above command worked, there is a way to improve the result. Since MyMix-2 workload contains several very different phases, it does not make much sense to calculate averages over the entire experiment. I want to use just the last phase called ``top2'' instead:
stan> cd /usr/local/polygraph/ReportGen/ stan> ./label_results --log_extractor_opts "--phases top2" \ test2 /tmp/logs/*log ./label_results: creating: `/tmp/polyrep/test2/clt.All.lx' ./label_results: creating: `/tmp/polyrep/test2/clt.Traces.lx'As you can see, I have instructed label_results to pass " --phases top2" options to lx which would limit the aggregate statistics to the specified phase. Note that trace graphs will be generated for the entire duration of the experiment, which is often what you want.
By now, you may get an idea that label_results does not know much about phases or even log files. That is true. All I was doing here was assigning a ``name'' to a particular snapshot of the results. Such a snapshot may include the entire experiment, one phase, or whatever the log extractor is capable of extracting.
Label_results just calls lx and ltrace with appropriate options and places the output into a well-known location, that's all. This divide-and-conquer principle may be useful if you want to use your own extraction tools or options: just use appropriate label_results command line options to configure log extractors.
After the results are labeled, I am ready to build reports.
Step 2: Making the reports
Since I named two versions of the results, I will build two reports:
stan> cd /usr/local/polygraph/ReportGen/ stan> ./make_report test1 ./make_report: creating: `/tmp/polyrep/test1/index.html' ./plot_traces: plotting: `/tmp/polyrep/test1/rates.png' ./plot_traces: plotting: `/tmp/polyrep/test1/rptms.png' ./plot_traces: plotting: `/tmp/polyrep/test1/hrs.png' ./plot_traces: plotting: `/tmp/polyrep/test1/xact_lvl.png' ./plot_traces: plotting: `/tmp/polyrep/test1/conn_lvl.png' ./plot_traces: plotting: `/tmp/polyrep/test1/errors.png' ./make_report: local report URL is file:/tmp/polyrep/test1/index.html stan> ./make_report test2 ./make_report: creating: `/tmp/polyrep/test2/index.html' ./plot_traces: plotting: `/tmp/polyrep/test2/rates.png' ./plot_traces: plotting: `/tmp/polyrep/test2/rptms.png' ./plot_traces: plotting: `/tmp/polyrep/test2/hrs.png' ./plot_traces: plotting: `/tmp/polyrep/test2/xact_lvl.png' ./plot_traces: plotting: `/tmp/polyrep/test2/conn_lvl.png' ./plot_traces: plotting: `/tmp/polyrep/test2/errors.png' ./make_report: local report URL is file:/tmp/polyrep/test2/index.htmlNote that make_report was calling plot_traces, another Report Generator tool. You can use plot_traces as a stand-alone application as well.
Viewing and sharing the results
Now I can either view local copies of the reports on stan (using a local browser and the ``file:'' URLs printed above), or I can copy the reports to a Web server. When copying, just replicate the entire directories, like this.
stan> scp -pr /tmp/polyrep/test1 \ web:/httpd/htdocs/Polygraph/polyrep/ stan> scp -pr /tmp/polyrep/test2 \ web:/httpd/htdocs/Polygraph/polyrep/
As with any automated report generator, caution should be taken when interpreting and sharing the results. Report Generator may not be able to correctly interpret all the ``interesting'' things you feed it with. For example, if you label a collection of client- and server-side logs (i.e., if you mix client- and server-side logs when labeling them), do not blame Report Generator for producing strange results.
Averaging over the entire duration of a test is often inappropriate. See examples above on how to label a particular phase instead.