Similar experience here, except that I've solved similar issues (and far more complicated issues) with DTrace instead of strace. The range of information that DTrace can retrieve about a running application surpasses anything that strace can retrieve. DTrace can retrieve information on a systemic scale, whereas each strace instance operates on a single process. And the overhead is significantly lower with DTrace.
For many of the systems applications I design for Illumos, I've used DTrace probes as a way of logging very frequent events, on demand (to avoid frequent IO). All of the events that _must_ be logged for the application to function properly, are logged and fsync'd.
I think that in most systems dynamic tracing will eventually replace a significant portion of the logging functionality that people code into their applications.
Either way, if you think strace is sweet, give DTrace a spin.
Some helpful links:
[0] A video demostration of DTrace, by the creator of DTrace.
[0] http://www.youtube.com/watch?v=6chLw2aodYQ
[1] A post I wrote, demonstrating DTrace. Similar posts can be found on the blog's dtrace-addict page.
[1] http://nickziv.wordpress.com/2011/04/08/adventures-of-a-dtra...
[2] A wiki that contains DTrace examples for various languages and system facilities. Some examples may be Illumos-centric.
[2] http://www.solarisinternals.com/wiki/index.php/DTrace_Topics
[3] DTrace's home. Contains blogs by the engineers behind DTrace.
[3] http://www.dtrace.org
UPDATE:
Meant to reply to parent's parent.
For many of the systems applications I design for Illumos, I've used DTrace probes as a way of logging very frequent events, on demand (to avoid frequent IO). All of the events that _must_ be logged for the application to function properly, are logged and fsync'd.
I think that in most systems dynamic tracing will eventually replace a significant portion of the logging functionality that people code into their applications.
Either way, if you think strace is sweet, give DTrace a spin.
Some helpful links:
[0] A video demostration of DTrace, by the creator of DTrace.
[0] http://www.youtube.com/watch?v=6chLw2aodYQ
[1] A post I wrote, demonstrating DTrace. Similar posts can be found on the blog's dtrace-addict page.
[1] http://nickziv.wordpress.com/2011/04/08/adventures-of-a-dtra...
[2] A wiki that contains DTrace examples for various languages and system facilities. Some examples may be Illumos-centric.
[2] http://www.solarisinternals.com/wiki/index.php/DTrace_Topics
[3] DTrace's home. Contains blogs by the engineers behind DTrace.
[3] http://www.dtrace.org
UPDATE: Meant to reply to parent's parent.