Commit fea69166 authored by Florent D'Halluin's avatar Florent D'Halluin
Browse files

Update benchmarks.

This is a compound patch that can hardly be divided into separate
patches.  It includes:

1. OpenFST integration.

  If OpenFST is present, OpenFST benchmarks are made for some
  algorithms.

2. Some (incomplete) documentation.

  Describe the benchmarking process, input automata, and results.

3. Clean up & name consistency.

  Move existing benchmarks into directories with consistent names
  (the benchmarked algorithms), store results in a specific
  directory according to input and other parameters.

Patch details:

1. OpenFST integration:

* configure.ac: Add --with-openfst option.
* src/bench/common/alpha.syms: Add missing line.
* src/bench/common/bench.mk: Fix -DVCSN_SRC_DIR flag.
* src/bench/common/bench_macros.hh: Add convenient macros

* src/bench/determinization/Makefile.bench: Add OpenFST bench.
* src/bench/determinization/determinization_aut_2n_openfst_bench.hh:
New (OpenFST benchmark program).

* src/bench/minimization/Makefile.bench: New (add OpenFST bench).
* src/bench/minimization/minimization_aut_2n_openfst_bench.hh:
New (OpenFST benchmark program).
* src/bench/minimization/minimization_aut_debruijn_openfst_bench.hh,
New (OpenFST benchmark program).

2. Some (incomplete) documentation:

* src/bench/README: New.  General information.
* src/bench/common/README: New.
* src/bench/common/README_AUTOMATA: New.

3. Clean up & name consistency:

* configure.ac: Add/remove makefiles.
* src/bench/Makefile.am: Add/remove directories.
* src/bench/generate_bench.sh: Add/remove directories.

* src/bench/common/Makefile.am: Clean up.
* src/bench/common/bench_automaton.hh: Remove (unused).
* src/bench/common/bench_constructs.hh: Clean up, add automata.

Move accessible to its own directory:

* src/bench/accessible/Makefile.bench: New.
* src/bench/iterators/accessible_bench.hh:
Rename into src/bench/accessible/accessible_bench.hh.

Clean up composition:

* src/bench/composition/Makefile.bench,
* src/bench/composition/normalized_composition_bench.hh: Consistency.
* src/bench/composition/rw_composition_bench.hh: Rename into
src/bench/composition/realtime_composition_bench.hh.

Clean up derived-terms:

* src/bench/derived_terms/Makefile.bench,
* src/bench/derived_terms/derived_terms_bench.hh: Consistency.

Clean up determinization:

* src/bench/determinization/Makefile.bench: Consistency.
* src/bench/determinization/determinization_bench.hh: Rename into
src/bench/determinization/determinization_aut_2n_bench.hh.
* src/bench/determinization/determinization_bench_fsm.cc:
Remove (unused).

Clean up eps_removal:

* src/bench/eps_removal/Makefile.bench,
* src/bench/eps_removal/eps_removal_bench.hh: Consistency.

Clean up iterators:

* src/bench/iterators/Makefile.bench,
* src/bench/iterators/iterator_bench.hh: Consistency.

Clean up quotient:

* src/bench/quotient/Makefile.bench,
* src/bench/quotient/quotient_bool_bench.hh,
* src/bench/quotient/quotient_z_bench.hh: Consistency.

Clean up minimization:

* src/bench/minimization/Makefile.bench: New (merge from
minimization_2n and minimization_debruijn).
* src/bench/minimization_2n/Makefile.bench: Remove (unused).
* src/bench/minimization_debruijn/Makefile.bench: Remove (unused).

* src/bench/minimization_2n/minimization_2n_brzozowski_bench.hh:
Rename into
src/bench/minimization/minimization_aut_2n_brzozowski_bench.hh.
* src/bench/minimization_2n/minimization_2n_hopcroft_bench.hh:
Rename into
src/bench/minimization/minimization_aut_2n_hopcroft_bench.hh.
* src/bench/minimization_2n/minimization_2n_moore_bench.hh:
Rename into
src/bench/minimization/minimization_aut_2n_moore_bench.hh.
* src/bench/minimization_debruijn/minimization_debruijn_brzozowski_bench.hh:
Rename into
src/bench/minimization/minimization_aut_debruijn_brzozowski_bench.hh.
* src/bench/minimization_debruijn/minimization_debruijn_hopcroft_bench.hh:
Rename into
src/bench/minimization/minimization_aut_debruijn_hopcroft_bench.hh.
* src/bench/minimization_debruijn/minimization_debruijn_moore_bench.hh:
Rename into
src/bench/minimization/minimization_aut_debruijn_moore_bench.hh.
parent 2be722fd
2009-06-17 d-halluin <d-halluin@lrde.epita.fr>
Update benchmarks.
This is a compound patch that can hardly be divided into separate
patches. It includes:
1. OpenFST integration.
If OpenFST is present, OpenFST benchmarks are made for some
algorithms.
2. Some (incomplete) documentation.
Describe the benchmarking process, input automata, and results.
3. Clean up & name consistency.
Move existing benchmarks into directories with consistent names
(the benchmarked algorithms), store results in a specific
directory according to input and other parameters.
Patch details:
1. OpenFST integration:
* configure.ac: Add --with-openfst option.
* src/bench/common/alpha.syms: Add missing line.
* src/bench/common/bench.mk: Fix -DVCSN_SRC_DIR flag.
* src/bench/common/bench_macros.hh: Add convenient macros
* src/bench/determinization/Makefile.bench: Add OpenFST bench.
* src/bench/determinization/determinization_aut_2n_openfst_bench.hh:
New (OpenFST benchmark program).
* src/bench/minimization/Makefile.bench: New (add OpenFST bench).
* src/bench/minimization/minimization_aut_2n_openfst_bench.hh:
New (OpenFST benchmark program).
* src/bench/minimization/minimization_aut_debruijn_openfst_bench.hh,
New (OpenFST benchmark program).
2. Some (incomplete) documentation:
* src/bench/README: New. General information.
* src/bench/common/README: New.
* src/bench/common/README_AUTOMATA: New.
3. Clean up & name consistency:
* configure.ac: Add/remove makefiles.
* src/bench/Makefile.am: Add/remove directories.
* src/bench/generate_bench.sh: Add/remove directories.
* src/bench/common/Makefile.am: Clean up.
* src/bench/common/bench_automaton.hh: Remove (unused).
* src/bench/common/bench_constructs.hh: Clean up, add automata.
Move accessible to its own directory:
* src/bench/accessible/Makefile.bench: New.
* src/bench/iterators/accessible_bench.hh:
Rename into src/bench/accessible/accessible_bench.hh.
Clean up composition:
* src/bench/composition/Makefile.bench,
* src/bench/composition/normalized_composition_bench.hh: Consistency.
* src/bench/composition/rw_composition_bench.hh: Rename into
src/bench/composition/realtime_composition_bench.hh.
Clean up derived-terms:
* src/bench/derived_terms/Makefile.bench,
* src/bench/derived_terms/derived_terms_bench.hh: Consistency.
Clean up determinization:
* src/bench/determinization/Makefile.bench: Consistency.
* src/bench/determinization/determinization_bench.hh: Rename into
src/bench/determinization/determinization_aut_2n_bench.hh.
* src/bench/determinization/determinization_bench_fsm.cc:
Remove (unused).
Clean up eps_removal:
* src/bench/eps_removal/Makefile.bench,
* src/bench/eps_removal/eps_removal_bench.hh: Consistency.
Clean up iterators:
* src/bench/iterators/Makefile.bench,
* src/bench/iterators/iterator_bench.hh: Consistency.
Clean up quotient:
* src/bench/quotient/Makefile.bench,
* src/bench/quotient/quotient_bool_bench.hh,
* src/bench/quotient/quotient_z_bench.hh: Consistency.
Clean up minimization:
* src/bench/minimization/Makefile.bench: New (merge from
minimization_2n and minimization_debruijn).
* src/bench/minimization_2n/Makefile.bench: Remove (unused).
* src/bench/minimization_debruijn/Makefile.bench: Remove (unused).
* src/bench/minimization_2n/minimization_2n_brzozowski_bench.hh:
Rename into
src/bench/minimization/minimization_aut_2n_brzozowski_bench.hh.
* src/bench/minimization_2n/minimization_2n_hopcroft_bench.hh:
Rename into
src/bench/minimization/minimization_aut_2n_hopcroft_bench.hh.
* src/bench/minimization_2n/minimization_2n_moore_bench.hh:
Rename into
src/bench/minimization/minimization_aut_2n_moore_bench.hh.
* src/bench/minimization_debruijn/minimization_debruijn_brzozowski_bench.hh:
Rename into
src/bench/minimization/minimization_aut_debruijn_brzozowski_bench.hh.
* src/bench/minimization_debruijn/minimization_debruijn_hopcroft_bench.hh:
Rename into
src/bench/minimization/minimization_aut_debruijn_hopcroft_bench.hh.
* src/bench/minimization_debruijn/minimization_debruijn_moore_bench.hh:
Rename into
src/bench/minimization/minimization_aut_debruijn_moore_bench.hh.
2009-06-17 d-halluin <d-halluin@lrde.epita.fr>
 
Update cbs.
......
......@@ -41,6 +41,21 @@ case $ac_default_graph_impl in
*) AC_ERROR([Unknown graph implementation: $ac_default_graph_impl]);;
esac
# Specify the path OpenFST binaries for benchmarks.
ac_openfst_path=/usr/local/bin # default install dir in OpenFST 1.1
AC_ARG_WITH([openfst],
[AC_HELP_STRING([--with-openfst@<:@=DIR@:>@],
[specify the path to OpenFST binaries for benchmarks])],
[ac_openfst_path="$withval"],[])
AM_CONDITIONAL([ENABLE_OPENFST], [test -e "${ac_openfst_path}/fstcompile"])
if test -e "${ac_openfst_path}/fstcompile"; then
AC_DEFINE_UNQUOTED([OPENFST_PATH], ["$ac_openfst_path"],
[Path to OpenFST binaries.])
else
AC_MSG_WARN([OpenFST binaries not found in ${ac_openfst_path}, OpenFST benchmarks are disabled.])
fi
# Some C and C++ code used.
AC_PROG_CXX
AC_PROG_CC
......@@ -282,8 +297,8 @@ AC_CONFIG_FILES([
src/bench/determinization/Makefile
src/bench/eps_removal/Makefile
src/bench/iterators/Makefile
src/bench/minimization_2n/Makefile
src/bench/minimization_debruijn/Makefile
src/bench/accessible/Makefile
src/bench/minimization/Makefile
src/bench/quotient/Makefile
tools/Makefile
include/Makefile
......
# src/bench/Makefile.am -*- Makefile -*-
#
# Vaucanson, a generic library for finite state machines.
# Vaucanson, a generic library for finite state machines.
# Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 The
# Vaucanson Group.
#
......@@ -16,17 +16,22 @@
#
SUBDIRS = common determinization \
minimization_debruijn minimization_2n derived_terms \
quotient composition eps_removal \
iterators
SUBDIRS = common determinization \
minimization derived_terms \
quotient composition eps_removal \
iterators accessible
BENCHDIRS = determinization minimization_debruijn \
derived_terms minimization_2n quotient \
composition eps_removal \
iterators
BENCHDIRS = determinization minimization \
derived_terms quotient \
composition eps_removal \
iterators accessible
bench:
for f in $(BENCHDIRS); do \
(cd $$f && $(MAKE) $(AM_MAKEFLAGS) bench || exit 1) \
done
bench-clean:
for f in $(BENCHDIRS); do \
(cd $$f && $(MAKE) $(AM_MAKEFLAGS) bench-clean || exit 1) \
done
Benchmarking system:
This file describes the benchmarking system in Vaucanson.
See common/README for file descriptions.
See common/README_AUTOMATA for automata descriptions.
See <algorithm>/README for information on specific benchmarks.
==================
-- Organization --
==================
Fixme.
=============
-- OpenFST --
=============
Fixme.
# accessible/Makefile.bench * -*- Makefile -*-
bench: all
@mkdir default; \
echo '=================='; \
echo '|accessible bench|'; \
echo '=================='; \
for n in 1000 2000 3000 4000 5000; do \
echo "----------------------For $$n:"; \
echo "-------------- accessible:"; \
./accessible_bench $$n; \
done
bench-clean:
rm -rf default
......@@ -14,8 +14,9 @@
//
// The Vaucanson Group consists of people listed in the `AUTHORS' file.
//
#ifndef BENCHS_ITERATORS_ACCESSIBLE_BENCH_HH
# define BENCHS_ITERATORS_ACCESSIBLE_BENCH_HH
#ifndef BENCHS_ACCESSIBLE_ACCESSIBLE_BENCH_HH
# define BENCHS_ACCESSIBLE_ACCESSIBLE_BENCH_HH
# define CONTEXT boolean_automaton
......@@ -29,11 +30,9 @@
using namespace vcsn;
using namespace vcsn::CONTEXT;
# include <common/bench_constructs.hh>
void accessible_bench(int n_states)
void accessible_bench(int n)
{
if (not n_states)
if (not n)
return;
AUTOMATON_TYPES_EXACT(automaton_t);
......@@ -44,7 +43,7 @@ void accessible_bench(int n_states)
automaton_t a = make_automaton(alpha);
std::vector<hstate_t> state_list;
for (int i = 0; i < n_states; ++i)
for (int i = 0; i < n; ++i)
state_list.push_back(a.add_state());
a.set_initial(a.get_state(0));
a.set_final(a.get_state(0));
......@@ -53,17 +52,17 @@ void accessible_bench(int n_states)
for (std::vector<hstate_t>::iterator j = state_list.begin(); j != state_list.end(); ++j)
a.add_spontaneous(*i, *j);
BENCH_START("accessible", "Vaucanson - accessible");
//automaton_t ret = accessible(a);
BENCH_START("Vaucanson accessible", "FIXME.");
//automaton_t ret = accessible(a);
accessible_states(a);
BENCH_STOP();
BENCH_PARAMETER("n_states", (long) n_states);
BENCH_PARAMETER("_n_", (long) n);
std::stringstream name;
name << "bench_accessible_" << n_states;
name << "default/bench_accessible_" << n;
BENCH_VCSN_SAVE_AND_PRINT(name.str());
}
#endif // ! BENCHS_ITERATORS_ACCESSIBLE_BENCH_HH
#endif // ! BENCHS_ACCESSIBLE_ACCESSIBLE_BENCH_HH
EXTRA_DIST = alpha.syms bench_automaton.hh \
EXTRA_DIST = alpha.syms \
bench_constructs.hh bench.mk \
bench_macros.hh
test:
echo hella
Contents:
README_AUTOMATA - Description of the benchmark input automata.
alpha.syms - Table of symbols to use with OpenFST.
bench_constructs.hh - Construction of test automata.
bench_macros.hh - Convenient macros.
This file describes the automata used for benchmarking.
Functions to create these automata are located in bench_constructs.hh.
============
-- aut_2n --
============
Boolean automaton on {a, b, c}.
Has n states in its original form and 2^n states once determinized.
==================
-- aut_debruijn --
==================
FIXME.
====================
-- aut_linear_eps --
====================
FIXME.
============
-- aut_ab --
============
FIXME.
......@@ -3,5 +3,5 @@
include $(top_srcdir)/vaucanson.mk
AM_CPPFLAGS += -I$(top_srcdir)/src/bench -I$(top_builddir)/src/bench
AM_CPPFLAGS += -DVCSN_SRC_DIR=\"'\$(top_srcdir)'\"
AM_CPPFLAGS += -DVCSN_SRC_DIR="\"$(top_srcdir)\""
AM_CPPFLAGS += -DNDEBUG -O3 -DVCSN_BENCH
// bench_automaton.hh: this file is part of the Vaucanson project.
//
// Vaucanson, a generic library for finite state machines.
//
// Copyright (C) 2005, 2006 The Vaucanson Group.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// The complete GNU General Public Licence Notice can be found as the
// `COPYING' file in the root directory.
//
// The Vaucanson Group consists of people listed in the `AUTHORS' file.
//
#ifndef VCSN_BENCHS_COMMON_BENCH_AUTOMATON_HH
# define VCSN_BENCHS_COMMON_BENCH_AUTOMATON_HH
#include <vaucanson/boolean_automaton.hh>
#include <vaucanson/algebra/implementation/predefs.hh>
namespace vcsn {
namespace bench_automaton {
using namespace vcsn;
using namespace vcsn::algebra;
using namespace vcsn::algebra::small_alpha_letter;
typedef polynom<WordValue, bool> series_set_elt_value_t;
typedef Series<NumericalSemiring, Words> series_set_t;
typedef Graph
<
labels_are_letters,
WordValue,
bool,
series_set_elt_value_t,
char,
NoTag,
NoTag>
automaton_impl_t;
typedef Element<Automata<series_set_t>, automaton_impl_t>
automaton_t;
AUTOMATON_TYPES_EXACT(automaton_t);
AUTOMATON_FREEMONOID_TYPES_EXACT(automaton_t);
template <class T>
automaton_t make_automaton(const T& alphabet);
template <class InputIterator>
automaton_t make_automaton(InputIterator begin,
InputIterator end);
template <class InputIterator>
automata_set_t new_automata_set(InputIterator begin,
InputIterator end)
{
alphabet_t alpha;
for (InputIterator e = begin; e != end; ++e)
alpha.insert(*e);
semiring_t semiring;
monoid_t freemonoid (alpha);
series_set_t series (semiring, freemonoid);
return automata_set_t (series);
}
template <class InputIterator>
automaton_t make_automaton(InputIterator begin,
InputIterator end)
{
return automaton_t (new_automata_set(begin, end));
}
template <class T>
automaton_t make_automaton(const T& alphabet)
{
return make_automaton(alphabet.begin(), alphabet.end());
}
} // bench_automaton
} // vcsn
#endif // ! VCSN_BENCHS_COMMON_BENCH_AUTOMATON_HH
......@@ -14,12 +14,21 @@
//
// The Vaucanson Group consists of people listed in the `AUTHORS' file.
//
#ifndef VCSN_BENCHS_COMMON_BENCH_CONSTRUCTS_HH
# define VCSN_BENCHS_COMMON_BENCH_CONSTRUCTS_HH
#ifndef VCSN_BENCH_COMMON_BENCH_CONSTRUCTS_HH
# define VCSN_BENCH_COMMON_BENCH_CONSTRUCTS_HH
/// Create a nice automaton which will have 2^n states once determinized.
void aut_2n(unsigned n, automaton_t& a)
/// Create an aut_2n automaton (has 2^n states once determinized).
/// See README_AUTOMATA for aut2n.
automaton_t aut_2n(unsigned n)
{
AUTOMATON_TYPES_EXACT(automaton_t);
alphabet_t alpha;
alpha.insert('a');
alpha.insert('b');
alpha.insert('c');
automaton_t a = make_automaton(alpha);
std::vector<hstate_t> c(n);
for (unsigned i = 0; i < n; ++i)
c[i] = a.add_state();
......@@ -34,27 +43,48 @@ void aut_2n(unsigned n, automaton_t& a)
}
a.set_initial(c[0]);
a.set_final(c[0]);
return a;
}
void debruijn(unsigned n_value, automaton_t& an)
/// Create a aut_debruijn automaton.
/// See README_AUTOMATA for debruijn.
automaton_t aut_debruijn(unsigned n)
{
unsigned n = 1 << n_value;
AUTOMATON_TYPES_EXACT(automaton_t);
alphabet_t alpha;
alpha.insert('a');
alpha.insert('b');
alpha.insert('c');
automaton_t a = make_automaton(alpha);
n = 1 << n;
for (unsigned i = 0; i < n; ++i)
an.add_state();
an.set_initial(0);
a.add_state();
a.set_initial(0);
for (unsigned i = 0; i < n; ++i)
{
an.add_letter_transition(i, (i*2)%n, 'a');
an.add_letter_transition(i, (i*2+1)%n, 'b');
a.add_letter_transition(i, (i * 2) % n, 'a');
a.add_letter_transition(i, (i * 2 + 1) % n, 'b');
if ((i << 1) & n)
an.set_final(i);
a.set_final(i);
}
return a;
}
/// Create a special automaton for benching eps_removal
void aut_linear_eps(unsigned n, automaton_t& a)
/// Create a aut_linear_eps automaton (for epsilon removal).
/// See README_AUTOMATA for aut_linear_eps.
automaton_t aut_linear_eps(unsigned n)
{
AUTOMATON_TYPES_EXACT(automaton_t);
alphabet_t alpha;
alpha.insert('a');
automaton_t a = make_automaton(alpha);
int size = n * 500;
std::vector<hstate_t> c(size);
......@@ -74,6 +104,41 @@ void aut_linear_eps(unsigned n, automaton_t& a)
a.add_letter_transition(c[i], c[(i+1) % size], 'a');
a.set_initial(c[0]);
a.set_final(c[size - 1]);
return a;
}
#endif // ! VCSN_BENCHS_COMMON_BENCH_CONSTRUCTS_HH
/// Create a autab automaton (for quotient).
/// See README_AUTOMATA for autab.
automaton_t aut_ab()
{
AUTOMATON_TYPES_EXACT(automaton_t);
alphabet_t alpha;
alpha.insert('a');
alpha.insert('b');
alpha.insert('c');
automaton_t a = make_automaton(alpha);
std::vector<hstate_t> c(2);
c[0] = a.add_state();
c[1] = a.add_state();
a.set_initial(c[0]);
a.set_final(c[1]);
a.add_letter_transition(c[0], c[0], 'a');
a.add_letter_transition(c[0], c[0], 'b');
a.add_letter_transition(c[0], c[1], 'b');
a.add_weighted_transition(c[1], c[1], 2, "a");
a.add_weighted_transition(c[1], c[1], 2, "b");
return a;
}
#endif // ! VCSN_BENCH_COMMON_BENCH_CONSTRUCTS_HH
......@@ -18,8 +18,19 @@
#ifndef VCSN_BENCHS_COMMON_BENCH_MACROS_HH
# define VCSN_BENCHS_COMMON_BENCH_MACROS_HH
# include <config.h>
# include <cbs/bench/bench_macros.hh>
# define BENCH_OPENFST_ISYMBOLS VCSN_SRC_DIR "/src/bench/common/alpha.syms"
# define BENCH_OPENFST_DUMP(Name, Automaton) \
do { \
std::ofstream dumpfile(Name.c_str()); \
fsm_dump(dumpfile, Automaton); \
dumpfile.close(); \
} while (0)
# define BENCH_VCSN_SAVE_AND_PRINT(Name) \
BENCH_SAVE(Name + ".xml", bench::Options()); \
BENCH_SAVE(Name + ".out", bench::Options(bench::Options::VE_MINIMAL, \
......
# composition/Makefile.bench * -*- Makefile -*-
bench: $(noinst_PROGRAMS)
@echo '==================='; \
echo '|Composition bench|'; \
echo '==================='; \
for n_value in $$(seq 20 40); do \
echo "----------------------For $$n_value:"; \
echo "--------------VCSN (Realtime):"; \
./rw_composition-bench $$n_value; \
echo "--------------VCSN (Normalized):"; \
./normalized_composition-bench $$n_value; \
bench: all
@mkdir normalized; \
mkdir realtime; \
echo '==================='; \
echo '|Composition bench|'; \
echo '==================='; \
for n in $$(seq 20 40); do \
echo "----------------------For $$n:"; \
echo "--------------VCSN (Realtime):"; \
./realtime_composition_bench $$n; \
echo "--------------VCSN (Normalized):"; \
./normalized_composition_bench $$n; \