Ask HN: What's your best C/C++ profiling tool, hints and best practice?

I frequently come across wanting to profile C++ on Linux. I have used perf a lot before but do not have vtune handy. I have dabbled around with poor man's profiler but that seems to get trickier with lots and lots of threads. What are your favorite outside-the-box approaches? How do you figure out contention or IO wait issues?