Commit fed5912d authored by Florent D'Halluin's avatar Florent D'Halluin Committed by Alexandre Duret-Lutz
Browse files

Update cbs.

CBS version: aef3055d95b926d6242ebbce64a128df044cbba2.
See cbs/Changelog.
parent 8227214d
2009-04-28 d-halluin <d-halluin@lrde.epita.fr>
Update cbs.
CBS version: aef3055d95b926d6242ebbce64a128df044cbba2.
See cbs/Changelog.
2009-04-16 d-halluin <d-halluin@lrde.epita.fr>
 
Fix make check.
......
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Update README_LIBBENCH figures.
* README_LIBBENCH: Update figures.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Add data from vcsn tests.
* demo/vcsn/vcsn-quotient-fix-1.data,
* demo/vcsn/vcsn-quotient-fix.data,
* demo/vcsn/vcsn-quotient-orig.data,
* demo/vcsn/vcsn-quotient.plot: New.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Add plotting helpers to timer-bench.
* demo/timer-bench/get-results.sh,
* demo/timer-bench/timer-many-tasks-memory.plot,
* demo/timer-bench/timer-many-tasks.plot,
* demo/timer-bench/timer-recursion-memory.plot,
* demo/timer-bench/timer-recursion-time.plot,
* demo/timer-bench/timer-scoped-memory.plot,
* demo/timer-bench/timer-scoped.plot: New.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Update comparaison demo.
* demo/comparison/main.cc: Update.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Add memplot bench demo.
* configure.ac: Add Makefile.
* demo/Makefile.am: Add subdir.
* demo/memplot-bench/Makefile.am: New.
* demo/memplot-bench/main.cc: New.
* demo/memplot-bench/get-results.sh,
* demo/memplot-bench/results-memory.plot,
* demo/memplot-bench/results-time.plot: New. Plotting helpers.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Add comparison demo for several profilers.
Works with CBS, gprof, kcachegrind.
* configure.ac: Add Makefile.
* demo/Makefile.am: Add subdir.
* demo/comparison/Makefile.am,
* demo/comparison/README,
* demo/comparison/main.cc: Add demo.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Remove obsolete demo scripts.
* demo/bench/demo.sh: Deleted.
* demo/bench/display.sh: Deleted.
* demo/bench/plot.sh: Deleted.
* demo/timer/demo.sh: Deleted.
* demo/timer/display.sh: Deleted.
* demo/timer/timer-demo.sh: Deleted.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
(Memplot) Fix measure.
* lib/memplot.cc: Remove loop.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
(Memplot) Remove brk measure.
* include/cbs/bench/memplot.hh,
* lib/memplot.cc: Remove memory_brk.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Update demos.
* demo/Makefile.am: Add new dir.
* demo/bench/main.cc: Add memplot, tweak.
* demo/memplot/Makefile.am,
* demo/memplot/main.cc: Add memplot demo.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Tweak timer main task name.
* lib/timer.cc: Add macro for name, change name.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Update TODO.
* TODO: Update.
2009-04-26 d-halluin <d-halluin@lrde.epita.fr>
Add memplot implementation.
* configure.ac: Add new Makefile.
* include/cbs/bench/memplot.hh: Update interface.
* lib/Makefile.am: Add new file.
* lib/bench.cc: Add implementation.
* lib/memplot.cc: Add implementation.
2009-04-12 d-halluin <d-halluin@lrde.epita.fr>
Add convenient overloads for result/parameter.
......
Libbench: measure performance of equiped C++ code.
Influence on equiped code (in time taken per 1 million measure):
## Influence on instrumented code performance ##
2Ghz Intel Centrino, 1GB RAM:
Measures made on a 2GHz Intel Centrino, 1GB RAM.
-- CPU Time measures (BENCH_TASK) --
For 1 million measures:
2.5s for 1 task (1.25s in parent, 1.25s in child).
+3.5s for 100k different tasks.
......@@ -12,9 +16,23 @@ i.e. using CBS to measure 100 different tasks for 100000 measures takes:
+ 100 000 * 3.5s / 1 000 000 * 100 / 100 000 (= 0.350ms)
total: ~250ms.
Use the timer-bench demo to compute those figures on another system.
-- Memory measures (BENCH_MEMPLOT) --
Time:
2s per 100 000 measures.
Memory:
10MB per 100 000 measures.
Typical usage:
-- Benchmarking on another machine --
Use the timer-bench and memplot-bench demos to compute those figures
on your system.
## Typical usage ##
(see demo/bench/main.cc)
......@@ -63,13 +81,13 @@ void do_some_tests()
________________________________________________________
Note:
Parameters and results are std::string.
Use std::stringstream to convert other values:
Parameters and results are std::string, long or double.
Use explicit cast or std::stringstream to convert other values:
#include <stringstream>
int n = 42;
std::stringstream n_str;
n_str << n;
BENCH_PARAMETER("n", n_str.str());
BENCH_PARAMETER("n (string)", n_str.str());
BENCH_PARAMETER("n (long)", (long) n));
To do:
- Implement memplot
- Add more system info (e.g. hostname)
- Allow bench loops
~ Allow bench loops:
CBS works with vcsn's global bencher.
~ Support Windows & Mac:
Write windows & mac versions of the timing primitives.
vcsn:
- Equip bench
- Gather xml files
- Generate html
- Equip bench:
Done, run make bench on a powerful machine
- Gather xml files:
Keep the directories as they are in make bench?
- Generate html:
Probably not necessary.
Extraction tool in bin/plot.pl
......@@ -44,8 +44,11 @@ AC_CONFIG_FILES([
lib/Makefile
include/Makefile
demo/Makefile
demo/comparison/Makefile
demo/timer/Makefile
demo/timer-bench/Makefile
demo/memplot-bench/Makefile
demo/memplot/Makefile
demo/bench/Makefile])
AC_OUTPUT
......@@ -14,4 +14,4 @@
include $(top_srcdir)/cbs.mk
SUBDIRS = timer bench timer-bench
SUBDIRS = timer bench timer-bench memplot comparison memplot-bench
#!/bin/sh
##
make && ./timer-demo && ./display.sh&
#!/bin/sh
##
dot -Tpng benchmark_1.dot -o benchmark_1.png
dot -Tpng benchmark_10.dot -o benchmark_10.png
dot -Tpng benchmark_100.dot -o benchmark_100.png
dot -Tpng benchmark_1000.dot -o benchmark_1000.png
dot -Tpng benchmark_10000.dot -o benchmark_10000.png
display benchmark_1.png &
display benchmark_10.png &
display benchmark_100.png &
display benchmark_1000.png &
display benchmark_10000.png &
......@@ -34,6 +34,7 @@
{ \
list.push_back(k); \
} \
BENCH_MEMPLOT("Memory consumption"); \
list.clear(); \
}
......@@ -57,36 +58,28 @@ int main()
for (int i = 0; i < n; ++i)
e();
CONSUME(130 + iter * 20)
CONSUME(10 + iter * 10)
BENCH_STOP();
{
std::stringstream parameter;
parameter << iter;
BENCH_PARAMETER("iteration", (long) iter);
BENCH_PARAMETER("iteration", parameter.str());
}
{
std::stringstream parameter;
parameter << n;
BENCH_PARAMETER("n", (long) n);
BENCH_PARAMETER("n", parameter.str());
std::stringstream name_str;
name_str << "benchmark_" << n;
std::string name = name_str.str();
std::string name = "benchmark_" + parameter.str();
BENCH_PRINT(bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_PRINT(bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_SAVE(name + ".out", bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_SAVE(name + ".xml", bench::Options());
BENCH_SAVE(name + ".dot", bench::Options(bench::Options::VE_MINIMAL,
bench::Options::FO_DOT));
}
BENCH_SAVE(name + ".out", bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_SAVE(name + ".xml", bench::Options());
BENCH_SAVE(name + ".dot", bench::Options(bench::Options::VE_MINIMAL,
bench::Options::FO_DOT));
}
}
......@@ -132,6 +125,7 @@ void e()
{
BENCH_TASK_SCOPED("Task E");
CONSUME(50)
WAIT(1)
d(0);
......
#!/bin/sh
cat *.xml |\
sed -n '
s/ <parameter name="[^"]*" value="\([0-9]*\)" \/>/\1/p
s/ <result name="[^"]*" value="\([0-9]*\)" \/>/\1/p
s/<\/bench>/NL/p
' |\
tr '\012' ' ' |\
sed '
{
s/ NL /\
/g
}' |\
sort -g > plot.plot
## CBS, a C++ benchmarking suite.
##
## Copyright (C) 2007, 2009 The CBS Authors.
##
## This program is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License
## as published by the Free Software Foundation; either version 2
## of the License, or (at your option) any later version.
##
## The complete GNU General Public Licence Notice can be found as the
## `COPYING' file in the root directory.
##
## The CBS Authors consists of people listed in the `AUTHORS' file.
include $(top_srcdir)/cbs.mk
noinst_PROGRAMS = demo-cbs demo-gprof demo-generic
demo_cbs_SOURCES = main.cc
demo_cbs_LDADD = $(top_builddir)/lib/libbench.la
demo_cbs_CPPFLAGS = $(AM_CPPFLAGS)
demo_gprof_SOURCES = main.cc
demo_gprof_LDADD =
demo_gprof_CPPFLAGS = $(AM_CPPFLAGS) -DBENCH_DISABLED -pg
demo_gprof_LDFLAGS = -pg
demo_generic_SOURCES = main.cc
demo_generic_LDADD =
demo_generic_CPPFLAGS = $(AM_CPPFLAGS) -DBENCH_DISABLED
-- Using CBS --
In build dir:
time ./demo-cbs <n>
-- Using gprof --
In build dir:
time ./demo-gprof <n>
gprof ./demo-gprof
Note: results seem wrong.
-- Using callgrind --
In build dir:
time valgrind --tool=callgrind --dump-every-bb=10000000 ./demo-generic <n>
kcachegrind
rm callgrind.*
Note: callgrind files would stack between multiple executions.
// main.cc: this file is part of the CBS project.
//
// CBS, a C++ benchmarking suite.
//
// Copyright (C) 2007, 2009 Florent D'Halluin.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// The complete GNU General Public Licence Notice can be found as the
// `COPYING' file in the root directory.
//
#include <iostream>
#include <sstream>
#include <fstream>
#include <list>
#include <cbs/bench/bench_macros.hh>
void parent(int);
void new_integer(int);
void list_push_back(int, std::list<int>& list);
void file_io(int);
int main(int argc, char** argv)
{
BENCH_START("Profiler comparison",
"A simple program that consumes time and memory\n"
"The parameter n defines the program complexity,\n"
"i.e. the time and memory taken.");
int n;
if (argc < 2)
{
std::cerr << "Usage: " << argv[0] << " <int benchmark_depth>" << std::endl;
return -1;
}
std::stringstream n_str(argv[1]);
n_str >> n;
if (n < 0 || n > 100)
{
std::cerr << "<benchmark_depth> must be between 0 and 100." << std::endl;
return -1;
}
std::cout << "Bench for n=" << n << "..." << std::endl;
parent(n);
std::cout << "Completed." << std::endl;
std::string name = "bench_" + std::string(argv[1]);
BENCH_PARAMETER("n", (long) n);
BENCH_STOP();
BENCH_PRINT(bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_SAVE(name + ".out", bench::Options(bench::Options::VE_FULL,
bench::Options::FO_TEXT));
BENCH_SAVE(name + ".xml", bench::Options());
BENCH_SAVE(name + ".dot", bench::Options(bench::Options::VE_FULL,
bench::Options::FO_DOT));
}
void parent(int n)
{
BENCH_TASK_SCOPED("parent()");
std::list<int> list;
for (int i = 0; i < n; ++i)
{
new_integer(n);
list_push_back(n, list);
list.clear();
std::stringstream i_str;
i_str << i;
BENCH_MEMPLOT("parent(" + i_str.str() + ")");
}
}
void new_integer(int n)
{
BENCH_TASK_SCOPED("new_integer()");
for (int i = 0; i < n * 100000; ++i)
{
int* a = new int(4);
++(*a);
delete a;
}
file_io(n);
}
void list_push_back(int n, std::list<int>& list)
{
BENCH_TASK_SCOPED("list_push_back()");
for (int i = 0; i < n * 100000; ++i)
{
list.push_back(i);
}
if (n > 0)
list_push_back(n - 1, list);
}
void file_io(int n)
{
BENCH_TASK_SCOPED("file_io()");
std::ofstream file;
for (int i = 0; i < n * 1000; ++i)
{
file.open("_dummy");
file.close();
}
if (n > 0)
new_integer(n - 1);
}
## CBS, a C++ benchmarking suite.
##
## Copyright (C) 2007, 2009 The CBS Authors.
##
## This program is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License
## as published by the Free Software Foundation; either version 2
## of the License, or (at your option) any later version.
##
## The complete GNU General Public Licence Notice can be found as the
## `COPYING' file in the root directory.
##
## The CBS Authors consists of people listed in the `AUTHORS' file.
include $(top_srcdir)/cbs.mk
noinst_PROGRAMS = demo-memplot-bench
demo_memplot_bench_SOURCES = main.cc
demo_memplot_bench_LDADD = $(top_builddir)/lib/libbench.la
demo_memplot_bench_CPPFLAGS = $(AM_CPPFLAGS)
#!/bin/sh
../../../bin/plot.pl -a memplot_*.xml > memplot-bench.data
// main.cc: this file is part of the CBS project.
//
// CBS, a C++ benchmarking suite.
//
// Copyright (C) 2007, 2009 Florent D'Halluin.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// The complete GNU General Public Licence Notice can be found as the
// `COPYING' file in the root directory.
//
#include <iostream>
#include <cbs/bench/timer.hh>
#include <cbs/bench/bench_macros.hh>
using namespace timer;
int main()
{
int n;
for (n = 10000; n < 500000; n *= 2)
{
BENCH_START("memplot", "Empty tasks.");
for (int i = 0; i < n; ++i)
{
BENCH_MEMPLOT("Empty task");
}
{
BENCH_STOP();
std::stringstream parameter;
parameter << n;
BENCH_PARAMETER("empty tasks", parameter.str());
std::string name = "memplot_" + parameter.str();
BENCH_PRINT(bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_SAVE(name + ".out", bench::Options(bench::Options::VE_NORMAL,
bench::Options::FO_TEXT,
0));
BENCH_SAVE(name + ".xml", bench::Options());
BENCH_SAVE(name + ".dot", bench::Options(bench::Options::VE_MINIMAL,
bench::Options::FO_DOT));
}
}
}
set title "Memory consumption of CBS Memplot (2GHz Intel Centrino, 1GB RAM)"
set xlabel "Number of measures"
set ylabel "Memory usage (MB)"
set key top left
plot 'memplot-bench.data' using ($1):($2/1024/1024) t "Memory consumption" smooth unique w linespoints pt 5 lc 1 lw 2
set title "Time consumption of CBS Memplot (2GHz Intel Centrino, 1GB RAM)"
set xlabel "Number of measures"
set ylabel "Time taken (ms)"
set key top left
plot 'memplot-bench.data' using ($1):($3) t "CPU time" smooth unique w linespoints pt 5 lc 1 lw 2, \
'memplot-bench.data' using ($1):($4) t "user" smooth unique w linespoints pt 6 lc 1, \
'memplot-bench.data' using ($1):($5) t "system" smooth unique w linespoints pt 3 lc 1
## CBS, a C++ benchmarking suite.
##
## Copyright (C) 2007, 2009 The CBS Authors.
##
## This program is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License