Commit f65c621a authored by Alexandre Duret-Lutz's avatar Alexandre Duret-Lutz
Browse files

ltlcross: report exit_status and exit_code columns in CSV and JSON

* src/bin/ltlcross.cc: Report exit_status and exit_code columns in CSV
and JSON files.  Also output lines for failed translations, and add
a --omit-missing option to disable that.  Move the time column right
after exit_status and exit_code.
* src/bin/man/ltlcross.x: Document each column of the output.
* bench/ltl2tgba/tools: Use the "{name}cmd" notation.
* bench/ltl2tgba/sum.py: Adjust to the new columns.
* bench/ltl2tgba/README: Update to point to the man page for a
description of the columns.
* bench/ltl2tgba/Makefile.am: Build results.pdf as said announced in
README.
* bench/spin13/html.bottom: Update code to ignore these two new
columns and lines with null values.
* src/tgbatest/ltlcross3.test: Add tests.
* doc/org/ltlcross.org: Adjust examples.
* NEWS: Mention this.
parent 686a4548
......@@ -7,7 +7,13 @@ New in spot 1.2a (not released)
ltlcross '{small} ltl2tgba -s --small %f >%N' ...
will run the command "ltl2tgba -s --small %f >%N", but only
print "small" in output files.
- ltlcross' CSV and JSON output now contains two additional
columns: exit_status and exit_code, used to report failures of
the translator. If the translation failed, only the time is
reported, and the rest of the statistics, which are missing,
area left empty (in CVS) or null (in JSON). A new option,
--omit-missing can be used to remove lines for failed
translations, and remove these two columns.
* Bug fixes:
- ltlcross' CSV output now stricly follows RFC 4180.
- ltlcross failed to report missing input or output escape sequences
......
......@@ -34,9 +34,11 @@ OUTLOG = $(OUTPUTS:=.log)
CLEANFILES = $(OUTCSV) $(OUTJSON) $(OUTLOG) \
results.pdf results.aux results.log results.tex
.PHONY = run
.PHONY = run json
run: $(OUTJSON)
run: results.pdf
json: $(OUTJSON)
deps = $(srcdir)/tools \
$(top_srcdir)/configure.ac \
......
......@@ -117,40 +117,16 @@ this benchmark.
Reading the summaries
=======================
The various outputs (CSV, JSON, our LaTeX) may use the following
column headers:
* input: formula translated (as a string in the CSV output, and as
an index into the input table in the JSON output)
* tool: tool used to translated this formula (idem)
* states: number of states of the resulting automaton
* edges: number of physical arcs between these states
* transitions: number of logical transitions from one state to the other
(for instance if the atomic propositions are 'a' and 'b', and
edge labeled by 'a' represents two transitions labeled by
'a&b' and 'a&!b')
* acc: number of acceptance sets used; it should always be one
in this automaton since we are producing (degeneralized)
Büchi automata.
* scc: number of strongly conncected components in the produced automaton
* nondetstates: number of nondeterministic states
* nondeterministic: 0 if the automaton is deterministic
(no nondeterministic state), 1 otherwise
* time: time required by the translation (although this is measured with
the highest-resolution clock available, it is "wall time", so it
can be affected by the machine's load).
* product_states: number of states in a product if the automaton with some
random Kripke structure (one unique Kripke structure is generated
for each formula and used with automata produced by all tools)
* product_transitions: number of transitions in that product
* product_scc: number of strongly connected componebts in that product
The various outputs (CSV, JSON, our LaTeX) use column headers
that are described in doc/userdoc/ltlcross.html and also
in the ltlcross man page.
The summary tables produced by sum.py accumulate all these results for
all formulae, tool by tool. They display an additional column, called
'count', giving the number of formulae successfully translated (the
missing formulae correspond to timeouts).
For all these values (except count), the sammler number the better.
For all these values (except count), smaller numbers are better.
More details about ltlcross (used to produce these outputs) can be
......
......@@ -47,23 +47,26 @@ def process_file(filename):
ncols = len(data['fields'])
ntools = len(data['tool'])
datacols = range(2, ncols)
datacols = range(4, ncols)
fields = { name:index for index,name in enumerate(data["fields"]) }
toolcol = fields['tool']
inputcol = fields['formula']
statescol = fields['states']
inputs = data["inputs"]
# Index results by tool, then input
results = { t:{} for t in range(0, ntools) }
for l in data["results"]:
if l[statescol] == None:
continue
results[l[toolcol]][l[inputcol]] = l
for i in range(0, ntools):
# Remove any leading directory, and trailing %...
name = data["tool"][i]
name = name[name.rfind('/', 0, name.find(' ')) + 1:]
data["tool"][i] = latex_escape(name[0:name.find('%')])
data["tool"][i] = latex_escape(name[0:(name+'%').find('%')])
timecol = fields['time']
......@@ -72,7 +75,7 @@ def process_file(filename):
\subsection*{Cumulative summary}''' % latex_escape(filename))
print('\\noindent\\begin{tabular}{l' + ('r' * (ncols - 1)) + '}\n',
" & ".join(rot(latex_escape(["tool", "count"] + data["fields"][2:]))),
" & ".join(rot(latex_escape(["tool", "count"] + data["fields"][4:]))),
"\\\\")
for i in range(0, ntools):
# Compute sums over each column.
......@@ -96,7 +99,6 @@ states and more transitions.
header += 'c'
header += '}'
statescol = fields['states']
transcol = fields['transitions']
print(header)
......
......@@ -25,17 +25,17 @@
set dummy
# Add third-party tools if they are available.
test -n "$SPIN" && set "$@" "$SPIN -f %s >%N"
test -n "$LTL2BA" && set "$@" "$LTL2BA -f %s >%N"
test -n "$LTL3BA" && set "$@" "$LTL3BA -f %s >%N" \
"$LTL3BA -M -f %s >%N" \
"$LTL3BA -S -f %s >%N" \
"$LTL3BA -S -M -f %s >%N"
test -n "$SPIN" && set "$@" "{spin} $SPIN -f %s >%N"
test -n "$LTL2BA" && set "$@" "{ltl2ba} $LTL2BA -f %s >%N"
test -n "$LTL3BA" && set "$@" "{ltl3ba} $LTL3BA -f %s >%N" \
"{ltl3ba-M} $LTL3BA -M -f %s >%N" \
"{ltl3ba-S} $LTL3BA -S -f %s >%N" \
"{ltl3ba-SM} $LTL3BA -S -M -f %s >%N"
# Use -s to output a neverclaim, like the other tools.
set "$@" \
"$LTL2TGBA --det -s %s >%N" \
"$LTL2TGBA --small -s %s >%N"
"{ltl2tgba-D} $LTL2TGBA --det -s %s >%N" \
"{ltl2tgba} $LTL2TGBA --small -s %s >%N"
# If you want to add your own tool, you can add it here.
# See 'man ltlcross' for the list of %-escapes you may use
......
......@@ -303,7 +303,7 @@ var tfield;
for (var m = 0; m < nfields; ++m)
{
if (m == ifield || m == tfield)
if (!is_datacol(m))
continue;
var tmp = [];
for (var t = 0; t < ntools; ++t)
......@@ -352,16 +352,29 @@ var tfield;
for (var c = 0; c < nfields; ++c)
{
var name = rawdata.fields[c];
results.addColumn('number', name, name);
if (name == 'exit_status')
results.addColumn('string', name, name);
else
results.addColumn('number', name, name);
fields[name] = c;
}
// FIXME: we should uses rawdata.inputs to set ifield and tfield
tfield = fields['tool'];
ifield = fields['formula'];
sfield = fields['states'];
esfield = fields['exit_status'];
ecfield = fields['exit_code'];
is_datacol = function(c) {
return (c != ifield && c != tfield && c != esfield && c != ecfield);
}
nresults = rawdata.results.length;
for (var r = 0; r < nresults; ++r)
{
var row = rawdata.results[r];
if (row[sfield] == null)
continue;
results.addRow(row);
hashres[[row[tfield],row[ifield]]] = row;
}
......@@ -412,7 +425,7 @@ var tfield;
var col = 2;
for (var c = 0; c < nfields; ++c)
{
if (c != ifield && c != tfield)
if (is_datacol(c))
{
aggreg.push({column:c, aggregation: google.visualization.data.sum, type: 'number'})
aggreg.push({column:c, aggregation: google.visualization.data.avg, type: 'number'})
......@@ -469,7 +482,7 @@ var tfield;
var pos = 3;
for (var m = 0; m < nfields; ++m)
{
if (m != ifield && m != tfield)
if (is_datacol(m))
{
var row = [rawdata.fields[m]];
var max = sumresults.getColumnRange(pos + 2)['max'];
......@@ -498,7 +511,7 @@ var tfield;
sel3.append($("<option>").attr('value', -1).text('(nothing)'));
for (var c = 0; c < nfields; ++c)
{
if (c != ifield && c != tfield)
if (is_datacol(c))
{
sel.append($("<option>").attr('value', c).text(rawdata.fields[c]));
sel2.append($("<option>").attr('value', c).text(rawdata.fields[c]));
......@@ -525,7 +538,7 @@ var tfield;
var first = true;
for (var c = 0; c < nfields; ++c)
{
if (c != ifield && c != tfield)
if (is_datacol(c))
{
sel.append($("<option>").attr('value', c).text(rawdata.fields[c]));
if (first)
......
This diff is collapsed.
......@@ -93,6 +93,7 @@ Exit status:\n\
#define OPT_PRODUCTS 9
#define OPT_COLOR 10
#define OPT_NOCOMP 11
#define OPT_OMIT 12
static const argp_option options[] =
{
......@@ -150,6 +151,8 @@ static const argp_option options[] =
"output statistics as JSON in FILENAME or on standard output", 0 },
{ "csv", OPT_CSV, "FILENAME", OPTION_ARG_OPTIONAL,
"output statistics as CSV in FILENAME or on standard output", 0 },
{ "omit-missing", OPT_OMIT, 0, 0,
"do not output statistics for timeouts or failed translations", 0 },
/**************************************************/
{ 0, 0, 0, 0, "Miscellaneous options:", -1 },
{ "color", OPT_COLOR, "WHEN", OPTION_ARG_OPTIONAL,
......@@ -166,7 +169,6 @@ const struct argp_child children[] =
{ 0, 0, 0, 0 }
};
enum color_type { color_never, color_always, color_if_tty };
static char const *const color_args[] =
......@@ -201,7 +203,7 @@ bool no_complement = false;
bool stop_on_error = false;
int seed = 0;
unsigned products = 1;
bool opt_omit = false;
struct translator_spec
{
......@@ -289,6 +291,9 @@ struct statistics
{
statistics()
: ok(false),
status_str(0),
status_code(0),
time(0),
states(0),
transitions(0),
acc(0),
......@@ -302,14 +307,18 @@ struct statistics
terminal_aut(false),
weak_aut(false),
strong_aut(false),
time(0),
product_states(0),
product_transitions(0),
product_scc(0)
{
}
// If OK is false, only the status_str, status_code, and time fields
// should be valid.
bool ok;
const char* status_str;
int status_code;
double time;
unsigned states;
unsigned edges;
unsigned transitions;
......@@ -324,15 +333,17 @@ struct statistics
bool terminal_aut;
bool weak_aut;
bool strong_aut;
double time;
double product_states;
double product_transitions;
double product_scc;
static void
fields(std::ostream& os)
fields(std::ostream& os, bool all)
{
os << ("\"states\","
if (all)
os << "\"exit_status\",\"exit_code\",";
os << ("\"time\","
"\"states\","
"\"edges\","
"\"transitions\","
"\"acc\","
......@@ -346,33 +357,53 @@ struct statistics
"\"terminal_aut\","
"\"weak_aut\","
"\"strong_aut\","
"\"time\","
"\"product_states\","
"\"product_transitions\","
"\"product_scc\"");
}
void
to_csv(std::ostream& os)
to_csv(std::ostream& os, bool all, const char* na = "")
{
os << states << ','
<< edges << ','
<< transitions << ','
<< acc << ','
<< scc << ','
<< nonacc_scc << ','
<< terminal_scc << ','
<< weak_scc << ','
<< strong_scc << ','
<< nondetstates << ','
<< nondeterministic << ','
<< terminal_aut << ','
<< weak_aut << ','
<< strong_aut << ','
<< time << ','
<< product_states << ','
<< product_transitions << ','
<< product_scc;
if (all)
os << '"' << status_str << "\"," << status_code << ',';
os << time << ',';
if (ok)
os << states << ','
<< edges << ','
<< transitions << ','
<< acc << ','
<< scc << ','
<< nonacc_scc << ','
<< terminal_scc << ','
<< weak_scc << ','
<< strong_scc << ','
<< nondetstates << ','
<< nondeterministic << ','
<< terminal_aut << ','
<< weak_aut << ','
<< strong_aut << ','
<< product_states << ','
<< product_transitions << ','
<< product_scc;
else
os << na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na << ','
<< na;
}
};
......@@ -470,6 +501,9 @@ parse_opt(int key, char* arg, struct argp_state*)
case OPT_NOCOMP:
no_complement = true;
break;
case OPT_OMIT:
opt_omit = true;
break;
case OPT_SEED:
seed = to_pos_int(arg);
break;
......@@ -784,27 +818,37 @@ namespace
int es = exec_with_timeout(cmd.c_str());
xtime_t after = gethrxtime();
const char* status_str = 0;
const spot::tgba* res = 0;
if (timed_out)
{
// This is not considered to be a global error.
std::cerr << "warning: timeout during execution of command\n";
++timeout_count;
status_str = "timeout";
es = -1;
}
else if (WIFSIGNALED(es))
{
status_str = "signal";
es = WTERMSIG(es);
global_error() << "error: execution terminated by signal "
<< WTERMSIG(es) << ".\n";
<< es << ".\n";
end_error();
}
else if (WIFEXITED(es) && WEXITSTATUS(es) != 0)
{
es = WEXITSTATUS(es);
status_str = "exit code";
global_error() << "error: execution returned exit code "
<< WEXITSTATUS(es) << ".\n";
<< es << ".\n";
end_error();
}
else
{
status_str = "ok";
es = 0;
switch (output.format)
{
case printable_result_filename::Spin:
......@@ -814,6 +858,8 @@ namespace
res = spot::neverclaim_parse(filename, pel, &dict);
if (!pel.empty())
{
status_str = "parse error";
es = -1;
std::ostream& err = global_error();
err << "error: failed to parse the produced neverclaim.\n";
spot::format_neverclaim_parse_errors(err, filename, pel);
......@@ -829,6 +875,8 @@ namespace
std::ifstream f(output.val()->name());
if (!f)
{
status_str = "no output";
es = -1;
global_error() << "Cannot open " << output.val()
<< std::endl;
end_error();
......@@ -838,6 +886,8 @@ namespace
res = spot::lbtt_parse(f, error, &dict);
if (!res)
{
status_str = "parse error";
es = -1;
global_error() << ("error: failed to parse output in "
"LBTT format: ")
<< error << std::endl;
......@@ -854,6 +904,8 @@ namespace
aut = spot::dstar_parse(filename, pel, &dict);
if (!pel.empty())
{
status_str = "parse error";
es = -1;
std::ostream& err = global_error();
err << "error: failed to parse the produced DSTAR"
" output.\n";
......@@ -873,41 +925,48 @@ namespace
assert(!"unreachable code");
}
}
// Compute statistics.
if (res && want_stats)
if (want_stats)
{
statistics* st = &(*fstats)[translator_num];
st->ok = true;
spot::tgba_sub_statistics s = sub_stats_reachable(res);
st->states = s.states;
st->edges = s.transitions;
st->transitions = s.sub_transitions;
st->acc = res->number_of_acceptance_conditions();
spot::scc_map m(res);
m.build_map();
unsigned c = m.scc_count();
st->scc = m.scc_count();
st->nondetstates = spot::count_nondet_states(res);
st->nondeterministic = st->nondetstates != 0;
for (unsigned n = 0; n < c; ++n)
st->status_str = status_str;
st->status_code = es;
double prec = XTIME_PRECISION;
st->time = (after - before) / prec;
// Compute statistics.
if (res)
{
if (!m.accepting(n))
++st->nonacc_scc;
else if (is_terminal_scc(m, n))
++st->terminal_scc;
else if (is_weak_scc(m, n))
++st->weak_scc;
st->ok = true;
spot::tgba_sub_statistics s = sub_stats_reachable(res);
st->states = s.states;
st->edges = s.transitions;
st->transitions = s.sub_transitions;
st->acc = res->number_of_acceptance_conditions();
spot::scc_map m(res);
m.build_map();
unsigned c = m.scc_count();
st->scc = m.scc_count();
st->nondetstates = spot::count_nondet_states(res);
st->nondeterministic = st->nondetstates != 0;
for (unsigned n = 0; n < c; ++n)
{
if (!m.accepting(n))
++st->nonacc_scc;
else if (is_terminal_scc(m, n))
++st->terminal_scc;
else if (is_weak_scc(m, n))
++st->weak_scc;
else
++st->strong_scc;
}
if (st->strong_scc)
st->strong_aut = true;
else if (st->weak_scc)
st->weak_aut = true;
else
++st->strong_scc;
st->terminal_aut = true;
}
if (st->strong_scc)
st->strong_aut = true;
else if (st->weak_scc)
st->weak_aut = true;
else
st->terminal_aut = true;
double prec = XTIME_PRECISION;
st->time = (after - before) / prec;
}
output.cleanup();
return res;
......@@ -1401,18 +1460,18 @@ print_stats_csv(const char* filename)
assert(rounds == formulas.size());
*out << "\"formula\",\"tool\",";
statistics::fields(*out);
statistics::fields(*out, !opt_omit);
*out << "\r\n";
for (unsigned r = 0; r < rounds; ++r)
for (unsigned t = 0; t < ntrans; ++t)
if (vstats[r][t].ok)
if (!opt_omit || vstats[r][t].ok)
{
*out << "\"";
spot::escape_rfc4180(*out, formulas[r]);
*out << "\",\"";
spot::escape_rfc4180(*out, translators[t].name);
*out << "\",";
vstats[r][t].to_csv(*out);
vstats[r][t].to_csv(*out, !opt_omit);
*out << "\r\n";
}
delete outfile;
......@@ -1453,19 +1512,19 @@ print_stats_json(const char* filename)
spot::escape_str(*out, formulas[r]);
}
*out << ("\"\n ],\n \"fields\": [\n \"formula\",\"tool\",");
statistics::fields(*out);
statistics::fields(*out, !opt_omit);
*out << "\n ],\n \"inputs\": [ 0, 1 ],";
*out << "\n \"results\": [";
bool notfirst = false;
for (unsigned r = 0; r < rounds; ++r)
for (unsigned t = 0; t < ntrans; ++t)
if (vstats[r][t].ok)
if (!opt_omit || vstats[r][t].ok)
{
if (notfirst)
*out << ',';
notfirst = true;
*out << "\n [ " << r << ',' << t << ',';
vstats[r][t].to_csv(*out);
vstats[r][t].to_csv(*out, !opt_omit, "null");
*out << " ]";
}
*out << "\n ]\n}\n";
......
......@@ -68,6 +68,110 @@ When this variable is defined, temporary files are not removed.
This is mostly useful for debugging.
[OUTPUT DATA]
The following columns are output in the CSV or JSON files.
.TP 7
\fBformula\fR
The formula translated.
.TP
\fBtool\fR
The tool used to translate this formula. This is either the value of the
full \fICOMMANDFMT\fR string specified on the command-line, or,
if \fICOMMANDFMT\fR has the form \f(CW{\fISHORTNAME\fR\f(CW}\fR\FiCMD\fR,
the value of \fISHORTNAME\fR.
.TP
\fBexit_status\fR, \fBexit_code\fR
Information about how the execution of the translator went. If the
option \fB\-\-omit\-missing\fR is given, these two columns are omitted
and only the lines corresponding to successful translation are output.
Otherwise, \fBexit_status\fR is a string that can take the following
values:
.RS
.TP
\f(CW"ok"\fR
The translator ran succesfully (this does not imply that the produced
automaton is correct) and ltlcross could parse the resulting
automaton. In this case \fBexit_code\fR is always 0.
.TP
\f(CW"timeout"\fR
The translator ran for more than the number of seconds
specified with the \fB\-\-timeout\fR option. In this
case \fBexit_code\fR is always -1.
.TP
\f(CW"exit code"\fR
The translator terminated with a non-zero exit code.