improve ltlcross & autcross benchmarking capabilities
-
When ltlcross (or autcross) runs several tools, it does so sequentially, in the order the tools were specified. To limit bias (caching issues, thermal throttling, ...) it might make sense to randomize the order in which tools are run. Let's add an option
--tool-order=random|same
that does that (random being the default). -
It would be nice to find a way to report memory usage in the CSV files. Maybe these tools could also use
wait4()
instead ofwaitpid()
where available to report a peak memory consumption? (I'm not sure ifwait4()
is enough when the children contains multiple processes, and I'm not sure how that combines withposix_spawn
.)