Commit f65c621a authored by Alexandre Duret-Lutz's avatar Alexandre Duret-Lutz
Browse files

ltlcross: report exit_status and exit_code columns in CSV and JSON

* src/bin/ltlcross.cc: Report exit_status and exit_code columns in CSV
and JSON files.  Also output lines for failed translations, and add
a --omit-missing option to disable that.  Move the time column right
after exit_status and exit_code.
* src/bin/man/ltlcross.x: Document each column of the output.
* bench/ltl2tgba/tools: Use the "{name}cmd" notation.
* bench/ltl2tgba/sum.py: Adjust to the new columns.
* bench/ltl2tgba/README: Update to point to the man page for a
description of the columns.
* bench/ltl2tgba/Makefile.am: Build results.pdf as said announced in
README.
* bench/spin13/html.bottom: Update code to ignore these two new
columns and lines with null values.
* src/tgbatest/ltlcross3.test: Add tests.
* doc/org/ltlcross.org: Adjust examples.
* NEWS: Mention this.
parent 686a4548
...@@ -7,7 +7,13 @@ New in spot 1.2a (not released) ...@@ -7,7 +7,13 @@ New in spot 1.2a (not released)
ltlcross '{small} ltl2tgba -s --small %f >%N' ... ltlcross '{small} ltl2tgba -s --small %f >%N' ...
will run the command "ltl2tgba -s --small %f >%N", but only will run the command "ltl2tgba -s --small %f >%N", but only
print "small" in output files. print "small" in output files.
- ltlcross' CSV and JSON output now contains two additional
columns: exit_status and exit_code, used to report failures of
the translator. If the translation failed, only the time is
reported, and the rest of the statistics, which are missing,
area left empty (in CVS) or null (in JSON). A new option,
--omit-missing can be used to remove lines for failed
translations, and remove these two columns.
* Bug fixes: * Bug fixes:
- ltlcross' CSV output now stricly follows RFC 4180. - ltlcross' CSV output now stricly follows RFC 4180.
- ltlcross failed to report missing input or output escape sequences - ltlcross failed to report missing input or output escape sequences
......
...@@ -34,9 +34,11 @@ OUTLOG = $(OUTPUTS:=.log) ...@@ -34,9 +34,11 @@ OUTLOG = $(OUTPUTS:=.log)
CLEANFILES = $(OUTCSV) $(OUTJSON) $(OUTLOG) \ CLEANFILES = $(OUTCSV) $(OUTJSON) $(OUTLOG) \
results.pdf results.aux results.log results.tex results.pdf results.aux results.log results.tex
.PHONY = run .PHONY = run json
run: $(OUTJSON) run: results.pdf
json: $(OUTJSON)
deps = $(srcdir)/tools \ deps = $(srcdir)/tools \
$(top_srcdir)/configure.ac \ $(top_srcdir)/configure.ac \
......
...@@ -117,40 +117,16 @@ this benchmark. ...@@ -117,40 +117,16 @@ this benchmark.
Reading the summaries Reading the summaries
======================= =======================
The various outputs (CSV, JSON, our LaTeX) may use the following The various outputs (CSV, JSON, our LaTeX) use column headers
column headers: that are described in doc/userdoc/ltlcross.html and also
in the ltlcross man page.
* input: formula translated (as a string in the CSV output, and as
an index into the input table in the JSON output)
* tool: tool used to translated this formula (idem)
* states: number of states of the resulting automaton
* edges: number of physical arcs between these states
* transitions: number of logical transitions from one state to the other
(for instance if the atomic propositions are 'a' and 'b', and
edge labeled by 'a' represents two transitions labeled by
'a&b' and 'a&!b')
* acc: number of acceptance sets used; it should always be one
in this automaton since we are producing (degeneralized)
Büchi automata.
* scc: number of strongly conncected components in the produced automaton
* nondetstates: number of nondeterministic states
* nondeterministic: 0 if the automaton is deterministic
(no nondeterministic state), 1 otherwise
* time: time required by the translation (although this is measured with
the highest-resolution clock available, it is "wall time", so it
can be affected by the machine's load).
* product_states: number of states in a product if the automaton with some
random Kripke structure (one unique Kripke structure is generated
for each formula and used with automata produced by all tools)
* product_transitions: number of transitions in that product
* product_scc: number of strongly connected componebts in that product
The summary tables produced by sum.py accumulate all these results for The summary tables produced by sum.py accumulate all these results for
all formulae, tool by tool. They display an additional column, called all formulae, tool by tool. They display an additional column, called
'count', giving the number of formulae successfully translated (the 'count', giving the number of formulae successfully translated (the
missing formulae correspond to timeouts). missing formulae correspond to timeouts).
For all these values (except count), the sammler number the better. For all these values (except count), smaller numbers are better.
More details about ltlcross (used to produce these outputs) can be More details about ltlcross (used to produce these outputs) can be
......
...@@ -47,23 +47,26 @@ def process_file(filename): ...@@ -47,23 +47,26 @@ def process_file(filename):
ncols = len(data['fields']) ncols = len(data['fields'])
ntools = len(data['tool']) ntools = len(data['tool'])
datacols = range(2, ncols) datacols = range(4, ncols)
fields = { name:index for index,name in enumerate(data["fields"]) } fields = { name:index for index,name in enumerate(data["fields"]) }
toolcol = fields['tool'] toolcol = fields['tool']
inputcol = fields['formula'] inputcol = fields['formula']
statescol = fields['states']
inputs = data["inputs"] inputs = data["inputs"]
# Index results by tool, then input # Index results by tool, then input
results = { t:{} for t in range(0, ntools) } results = { t:{} for t in range(0, ntools) }
for l in data["results"]: for l in data["results"]:
if l[statescol] == None:
continue
results[l[toolcol]][l[inputcol]] = l results[l[toolcol]][l[inputcol]] = l
for i in range(0, ntools): for i in range(0, ntools):
# Remove any leading directory, and trailing %... # Remove any leading directory, and trailing %...
name = data["tool"][i] name = data["tool"][i]
name = name[name.rfind('/', 0, name.find(' ')) + 1:] name = name[name.rfind('/', 0, name.find(' ')) + 1:]
data["tool"][i] = latex_escape(name[0:name.find('%')]) data["tool"][i] = latex_escape(name[0:(name+'%').find('%')])
timecol = fields['time'] timecol = fields['time']
...@@ -72,7 +75,7 @@ def process_file(filename): ...@@ -72,7 +75,7 @@ def process_file(filename):
\subsection*{Cumulative summary}''' % latex_escape(filename)) \subsection*{Cumulative summary}''' % latex_escape(filename))
print('\\noindent\\begin{tabular}{l' + ('r' * (ncols - 1)) + '}\n', print('\\noindent\\begin{tabular}{l' + ('r' * (ncols - 1)) + '}\n',
" & ".join(rot(latex_escape(["tool", "count"] + data["fields"][2:]))), " & ".join(rot(latex_escape(["tool", "count"] + data["fields"][4:]))),
"\\\\") "\\\\")
for i in range(0, ntools): for i in range(0, ntools):
# Compute sums over each column. # Compute sums over each column.
...@@ -96,7 +99,6 @@ states and more transitions. ...@@ -96,7 +99,6 @@ states and more transitions.
header += 'c' header += 'c'
header += '}' header += '}'
statescol = fields['states']
transcol = fields['transitions'] transcol = fields['transitions']
print(header) print(header)
......
...@@ -25,17 +25,17 @@ ...@@ -25,17 +25,17 @@
set dummy set dummy
# Add third-party tools if they are available. # Add third-party tools if they are available.
test -n "$SPIN" && set "$@" "$SPIN -f %s >%N" test -n "$SPIN" && set "$@" "{spin} $SPIN -f %s >%N"
test -n "$LTL2BA" && set "$@" "$LTL2BA -f %s >%N" test -n "$LTL2BA" && set "$@" "{ltl2ba} $LTL2BA -f %s >%N"
test -n "$LTL3BA" && set "$@" "$LTL3BA -f %s >%N" \ test -n "$LTL3BA" && set "$@" "{ltl3ba} $LTL3BA -f %s >%N" \
"$LTL3BA -M -f %s >%N" \ "{ltl3ba-M} $LTL3BA -M -f %s >%N" \
"$LTL3BA -S -f %s >%N" \ "{ltl3ba-S} $LTL3BA -S -f %s >%N" \
"$LTL3BA -S -M -f %s >%N" "{ltl3ba-SM} $LTL3BA -S -M -f %s >%N"
# Use -s to output a neverclaim, like the other tools. # Use -s to output a neverclaim, like the other tools.
set "$@" \ set "$@" \
"$LTL2TGBA --det -s %s >%N" \ "{ltl2tgba-D} $LTL2TGBA --det -s %s >%N" \
"$LTL2TGBA --small -s %s >%N" "{ltl2tgba} $LTL2TGBA --small -s %s >%N"
# If you want to add your own tool, you can add it here. # If you want to add your own tool, you can add it here.
# See 'man ltlcross' for the list of %-escapes you may use # See 'man ltlcross' for the list of %-escapes you may use
......
...@@ -303,7 +303,7 @@ var tfield; ...@@ -303,7 +303,7 @@ var tfield;
for (var m = 0; m < nfields; ++m) for (var m = 0; m < nfields; ++m)
{ {
if (m == ifield || m == tfield) if (!is_datacol(m))
continue; continue;
var tmp = []; var tmp = [];
for (var t = 0; t < ntools; ++t) for (var t = 0; t < ntools; ++t)
...@@ -352,16 +352,29 @@ var tfield; ...@@ -352,16 +352,29 @@ var tfield;
for (var c = 0; c < nfields; ++c) for (var c = 0; c < nfields; ++c)
{ {
var name = rawdata.fields[c]; var name = rawdata.fields[c];
results.addColumn('number', name, name); if (name == 'exit_status')
results.addColumn('string', name, name);
else
results.addColumn('number', name, name);
fields[name] = c; fields[name] = c;
} }
// FIXME: we should uses rawdata.inputs to set ifield and tfield // FIXME: we should uses rawdata.inputs to set ifield and tfield
tfield = fields['tool']; tfield = fields['tool'];
ifield = fields['formula']; ifield = fields['formula'];
sfield = fields['states'];
esfield = fields['exit_status'];
ecfield = fields['exit_code'];
is_datacol = function(c) {
return (c != ifield && c != tfield && c != esfield && c != ecfield);
}
nresults = rawdata.results.length; nresults = rawdata.results.length;
for (var r = 0; r < nresults; ++r) for (var r = 0; r < nresults; ++r)
{ {
var row = rawdata.results[r]; var row = rawdata.results[r];
if (row[sfield] == null)
continue;
results.addRow(row); results.addRow(row);
hashres[[row[tfield],row[ifield]]] = row; hashres[[row[tfield],row[ifield]]] = row;
} }
...@@ -412,7 +425,7 @@ var tfield; ...@@ -412,7 +425,7 @@ var tfield;
var col = 2; var col = 2;
for (var c = 0; c < nfields; ++c) for (var c = 0; c < nfields; ++c)
{ {
if (c != ifield && c != tfield) if (is_datacol(c))
{ {
aggreg.push({column:c, aggregation: google.visualization.data.sum, type: 'number'}) aggreg.push({column:c, aggregation: google.visualization.data.sum, type: 'number'})
aggreg.push({column:c, aggregation: google.visualization.data.avg, type: 'number'}) aggreg.push({column:c, aggregation: google.visualization.data.avg, type: 'number'})
...@@ -469,7 +482,7 @@ var tfield; ...@@ -469,7 +482,7 @@ var tfield;
var pos = 3; var pos = 3;
for (var m = 0; m < nfields; ++m) for (var m = 0; m < nfields; ++m)
{ {
if (m != ifield && m != tfield) if (is_datacol(m))
{ {
var row = [rawdata.fields[m]]; var row = [rawdata.fields[m]];
var max = sumresults.getColumnRange(pos + 2)['max']; var max = sumresults.getColumnRange(pos + 2)['max'];
...@@ -498,7 +511,7 @@ var tfield; ...@@ -498,7 +511,7 @@ var tfield;
sel3.append($("<option>").attr('value', -1).text('(nothing)')); sel3.append($("<option>").attr('value', -1).text('(nothing)'));
for (var c = 0; c < nfields; ++c) for (var c = 0; c < nfields; ++c)
{ {
if (c != ifield && c != tfield) if (is_datacol(c))
{ {
sel.append($("<option>").attr('value', c).text(rawdata.fields[c])); sel.append($("<option>").attr('value', c).text(rawdata.fields[c]));
sel2.append($("<option>").attr('value', c).text(rawdata.fields[c])); sel2.append($("<option>").attr('value', c).text(rawdata.fields[c]));
...@@ -525,7 +538,7 @@ var tfield; ...@@ -525,7 +538,7 @@ var tfield;
var first = true; var first = true;
for (var c = 0; c < nfields; ++c) for (var c = 0; c < nfields; ++c)
{ {
if (c != ifield && c != tfield) if (is_datacol(c))
{ {
sel.append($("<option>").attr('value', c).text(rawdata.fields[c])); sel.append($("<option>").attr('value', c).text(rawdata.fields[c]));
if (first) if (first)
......
This diff is collapsed.
...@@ -93,6 +93,7 @@ Exit status:\n\ ...@@ -93,6 +93,7 @@ Exit status:\n\
#define OPT_PRODUCTS 9 #define OPT_PRODUCTS 9
#define OPT_COLOR 10 #define OPT_COLOR 10
#define OPT_NOCOMP 11 #define OPT_NOCOMP 11
#define OPT_OMIT 12
static const argp_option options[] = static const argp_option options[] =
{ {
...@@ -150,6 +151,8 @@ static const argp_option options[] = ...@@ -150,6 +151,8 @@ static const argp_option options[] =
"output statistics as JSON in FILENAME or on standard output", 0 }, "output statistics as JSON in FILENAME or on standard output", 0 },
{ "csv", OPT_CSV, "FILENAME", OPTION_ARG_OPTIONAL, { "csv", OPT_CSV, "FILENAME", OPTION_ARG_OPTIONAL,
"output statistics as CSV in FILENAME or on standard output", 0 }, "output statistics as CSV in FILENAME or on standard output", 0 },
{ "omit-missing", OPT_OMIT, 0, 0,
"do not output statistics for timeouts or failed translations", 0 },
/**************************************************/ /**************************************************/
{ 0, 0, 0, 0, "Miscellaneous options:", -1 }, { 0, 0, 0, 0, "Miscellaneous options:", -1 },
{ "color", OPT_COLOR, "WHEN", OPTION_ARG_OPTIONAL, { "color", OPT_COLOR, "WHEN", OPTION_ARG_OPTIONAL,
...@@ -166,7 +169,6 @@ const struct argp_child children[] = ...@@ -166,7 +169,6 @@ const struct argp_child children[] =
{ 0, 0, 0, 0 } { 0, 0, 0, 0 }
}; };
enum color_type { color_never, color_always, color_if_tty }; enum color_type { color_never, color_always, color_if_tty };
static char const *const color_args[] = static char const *const color_args[] =
...@@ -201,7 +203,7 @@ bool no_complement = false; ...@@ -201,7 +203,7 @@ bool no_complement = false;
bool stop_on_error = false; bool stop_on_error = false;
int seed = 0; int seed = 0;
unsigned products = 1; unsigned products = 1;
bool opt_omit = false;
struct translator_spec struct translator_spec
{ {
...@@ -289,6 +291,9 @@ struct statistics ...@@ -289,6 +291,9 @@ struct statistics
{ {
statistics() statistics()
: ok(false), : ok(false),
status_str(0),
status_code(0),
time(0),
states(0), states(0),
transitions(0), transitions(0),
acc(0), acc(0),
...@@ -302,14 +307,18 @@ struct statistics ...@@ -302,14 +307,18 @@ struct statistics
terminal_aut(false), terminal_aut(false),
weak_aut(false), weak_aut(false),
strong_aut(false), strong_aut(false),
time(0),
product_states(0), product_states(0),
product_transitions(0), product_transitions(0),
product_scc(0) product_scc(0)
{ {
} }
// If OK is false, only the status_str, status_code, and time fields
// should be valid.
bool ok; bool ok;
const char* status_str;
int status_code;
double time;
unsigned states; unsigned states;
unsigned edges; unsigned edges;
unsigned transitions; unsigned transitions;
...@@ -324,15 +333,17 @@ struct statistics ...@@ -324,15 +333,17 @@ struct statistics
bool terminal_aut; bool terminal_aut;
bool weak_aut; bool weak_aut;
bool strong_aut; bool strong_aut;
double time;
double product_states; double product_states;
double product_transitions; double product_transitions;
double product_scc; double product_scc;
static void static void
fields(std::ostream& os) fields(std::ostream& os, bool all)
{ {
os << ("\"states\"," if (all)
os << "\"exit_status\",\"exit_code\",";
os << ("\"time\","
"\"states\","
"\"edges\"," "\"edges\","
"\"transitions\"," "\"transitions\","
"\"acc\"," "\"acc\","
...@@ -346,33 +357,53 @@ struct statistics ...@@ -346,33 +357,53 @@ struct statistics
"\"terminal_aut\"," "\"terminal_aut\","
"\"weak_aut\"," "\"weak_aut\","
"\"strong_aut\"," "\"strong_aut\","
"\"time\","
"\"product_states\"," "\"product_states\","
"\"product_transitions\"," "\"product_transitions\","
"\"product_scc\""); "\"product_scc\"");
} }
void void
to_csv(std::ostream& os) to_csv(std::ostream& os, bool all, const char* na = "")
{ {
os << states << ',' if (all)
<< edges << ',' os << '"' << status_str << "\"," << status_code << ',';
<< transitions << ',' os << time << ',';
<< acc << ',' if (ok)
<< scc << ',' os << states << ','
<< nonacc_scc << ',' << edges << ','
<< terminal_scc << ',' << transitions << ','
<< weak_scc << ',' << acc << ','
<< strong_scc << ',' << scc << ','
<< nondetstates << ',' << nonacc_scc << ','
<< nondeterministic << ',' << terminal_scc << ','
<< terminal_aut << ',' << weak_scc << ','
<< weak_aut << ',' << strong_scc << ','
<< strong_aut << ',' << nondetstates << ','
<< time << ',' << nondeterministic << ','
<< product_states << ',' << terminal_aut << ','
<< product_transitions << ',' << weak_aut << ','
<< product_scc; << strong_aut << ','
<< product_states << ','
<< product_transitions << ','
<< product_scc;
else
os << na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na;
} }
}; };
...@@ -470,6 +501,9 @@ parse_opt(int key, char* arg, struct argp_state*) ...@@ -470,6 +501,9 @@ parse_opt(int key, char* arg, struct argp_state*)
case OPT_NOCOMP: case OPT_NOCOMP:
no_complement = true; no_complement = true;
break; break;
case OPT_OMIT:
opt_omit = true;
break;
case OPT_SEED: case OPT_SEED:
seed = to_pos_int(arg); seed = to_pos_int(arg);
break; break;
...@@ -784,27 +818,37 @@ namespace ...@@ -784,27 +818,37 @@ namespace
int es = exec_with_timeout(cmd.c_str()); int es = exec_with_timeout(cmd.c_str());
xtime_t after = gethrxtime(); xtime_t after = gethrxtime();
const char* status_str = 0;
const spot::tgba* res = 0; const spot::tgba* res = 0;
if (timed_out) if (timed_out)
{ {
// This is not considered to be a global error. // This is not considered to be a global error.
std::cerr << "warning: timeout during execution of command\n"; std::cerr << "warning: timeout during execution of command\n";
++timeout_count; ++timeout_count;
status_str = "timeout";
es = -1;
} }
else if (WIFSIGNALED(es)) else if (WIFSIGNALED(es))
{ {
status_str = "signal";
es = WTERMSIG(es);
global_error() << "error: execution terminated by signal " global_error() << "error: execution terminated by signal "
<< WTERMSIG(es) << ".\n"; << es << ".\n";
end_error(); end_error();
} }
else if (WIFEXITED(es) && WEXITSTATUS(es) != 0) else if (WIFEXITED(es) && WEXITSTATUS(es) != 0)
{ {
es = WEXITSTATUS(es);
status_str = "exit code";
global_error() << "error: execution returned exit code " global_error() << "error: execution returned exit code "
<< WEXITSTATUS(es) << ".\n"; << es << ".\n";
end_error(); end_error();
} }
else else
{ {
status_str = "ok";
es = 0;
switch (output.format) switch (output.format)
{ {
case printable_result_filename::Spin: case printable_result_filename::Spin:
...@@ -814,6 +858,8 @@ namespace ...@@ -814,6 +858,8 @@ namespace
res = spot::neverclaim_parse(filename, pel, &dict); res = spot::neverclaim_parse(filename, pel, &dict);
if (!pel.empty()) if (!pel.empty())
{ {
status_str = "parse error";
es = -1;
std::ostream& err = global_error(); std::ostream& err = global_error();
err << "error: failed to parse the produced neverclaim.\n"; err << "error: failed to parse the produced neverclaim.\n";
spot::format_neverclaim_parse_errors(err, filename, pel); spot::format_neverclaim_parse_errors(err, filename, pel);
...@@ -829,6 +875,8 @@ namespace ...@@ -829,6 +875,8 @@ namespace
std::ifstream f(output.val()->name()); std::ifstream f(output.val()->name());
if (!f) if (!f)
{ {
status_str = "no output";
es = -1;
global_error() << "Cannot open " << output.val() global_error() << "Cannot open " << output.val()
<< std::endl; << std::endl;
end_error(); end_error();
...@@ -838,6 +886,8 @@ namespace ...@@ -838,6 +886,8 @@ namespace
res = spot::lbtt_parse(f, error, &dict); res = spot::lbtt_parse(f, error, &dict);
if (!res) if (!res)
{ {
status_str = "parse error";
es = -1;
global_error() << ("error: failed to parse output in " global_error() << ("error: failed to parse output in "
"LBTT format: ") "LBTT format: ")
<< error << std::endl; << error << std::endl;
...@@ -854,6 +904,8 @@ namespace ...@@ -854,6 +904,8 @@ namespace
aut = spot::dstar_parse