Enhanced PHP File Manager

⬆️ Go Up (..)

📊 Server Information

Server:

nginx/1.28.0

System:

Linux 6.8.0-90-generic

PHP Version:

8.4.15

User:

alqudsvolunteers

Directory:

/home/alqudsvolunteers/rootlink/proc/thread-self/root/usr/share/doc/trace-cmd/trace-cmd

Free Space:

80.3 GB

🔧 Quick Actions

✏️ Editing: trace-cmd-profile.1.html

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 10.2.0" />
<title>TRACE-CMD-PROFILE(1)</title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */

/* Default font. */
body {
  font-family: Georgia,serif;
}

/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
  font-family: Arial,Helvetica,sans-serif;
}

body {
  margin: 1em 5% 1em 5%;
}

a {
  color: blue;
  text-decoration: underline;
}
a:visited {
  color: fuchsia;
}

em {
  font-style: italic;
  color: navy;
}

strong {
  font-weight: bold;
  color: #083194;
}

h1, h2, h3, h4, h5, h6 {
  color: #527bbd;
  margin-top: 1.2em;
  margin-bottom: 0.5em;
  line-height: 1.3;
}

h1, h2, h3 {
  border-bottom: 2px solid silver;
}
h2 {
  padding-top: 0.5em;
}
h3 {
  float: left;
}
h3 + * {
  clear: left;
}
h5 {
  font-size: 1.0em;
}

div.sectionbody {
  margin-left: 0;
}

hr {
  border: 1px solid silver;
}

p {
  margin-top: 0.5em;
  margin-bottom: 0.5em;
}

ul, ol, li > p {
  margin-top: 0;
}
ul > li     { color: #aaa; }
ul > li > * { color: black; }

.monospaced, code, pre {
  font-family: "Courier New", Courier, monospace;
  font-size: inherit;
  color: navy;
  padding: 0;
  margin: 0;
}
pre {
  white-space: pre-wrap;
}

#author {
  color: #527bbd;
  font-weight: bold;
  font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}

#footer {
  font-size: small;
  border-top: 2px solid silver;
  padding-top: 0.5em;
  margin-top: 4.0em;
}
#footer-text {
  float: left;
  padding-bottom: 0.5em;
}
#footer-badges {
  float: right;
  padding-bottom: 0.5em;
}

#preamble {
  margin-top: 1.5em;
  margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
div.admonitionblock {
  margin-top: 2.0em;
  margin-bottom: 2.0em;
  margin-right: 10%;
  color: #606060;
}

div.content { /* Block element content. */
  padding: 0;
}

/* Block element titles. */
div.title, caption.title {
  color: #527bbd;
  font-weight: bold;
  text-align: left;
  margin-top: 1.0em;
  margin-bottom: 0.5em;
}
div.title + * {
  margin-top: 0;
}

td div.title:first-child {
  margin-top: 0.0em;
}
div.content div.title:first-child {
  margin-top: 0.0em;
}
div.content + div.title {
  margin-top: 0.0em;
}

div.sidebarblock > div.content {
  background: #ffffee;
  border: 1px solid #dddddd;
  border-left: 4px solid #f0f0f0;
  padding: 0.5em;
}

div.listingblock > div.content {
  border: 1px solid #dddddd;
  border-left: 5px solid #f0f0f0;
  background: #f8f8f8;
  padding: 0.5em;
}

div.quoteblock, div.verseblock {
  padding-left: 1.0em;
  margin-left: 1.0em;
  margin-right: 10%;
  border-left: 5px solid #f0f0f0;
  color: #888;
}

div.quoteblock > div.attribution {
  padding-top: 0.5em;
  text-align: right;
}

div.verseblock > pre.content {
  font-family: inherit;
  font-size: inherit;
}
div.verseblock > div.attribution {
  padding-top: 0.75em;
  text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
  text-align: left;
}

div.admonitionblock .icon {
  vertical-align: top;
  font-size: 1.1em;
  font-weight: bold;
  text-decoration: underline;
  color: #527bbd;
  padding-right: 0.5em;
}
div.admonitionblock td.content {
  padding-left: 0.5em;
  border-left: 3px solid #dddddd;
}

div.exampleblock > div.content {
  border-left: 3px solid #dddddd;
  padding-left: 0.5em;
}

div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; vertical-align: text-bottom; }
a.image:visited { color: white; }

dl {
  margin-top: 0.8em;
  margin-bottom: 0.8em;
}
dt {
  margin-top: 0.5em;
  margin-bottom: 0;
  font-style: normal;
  color: navy;
}
dd > *:first-child {
  margin-top: 0.1em;
}

ul, ol {
    list-style-position: outside;
}
ol.arabic {
  list-style-type: decimal;
}
ol.loweralpha {
  list-style-type: lower-alpha;
}
ol.upperalpha {
  list-style-type: upper-alpha;
}
ol.lowerroman {
  list-style-type: lower-roman;
}
ol.upperroman {
  list-style-type: upper-roman;
}

div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
  margin-top: 0.1em;
  margin-bottom: 0.1em;
}

tfoot {
  font-weight: bold;
}
td > div.verse {
  white-space: pre;
}

div.hdlist {
  margin-top: 0.8em;
  margin-bottom: 0.8em;
}
div.hdlist tr {
  padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
  font-weight: bold;
}
td.hdlist1 {
  vertical-align: top;
  font-style: normal;
  padding-right: 0.8em;
  color: navy;
}
td.hdlist2 {
  vertical-align: top;
}
div.hdlist.compact tr {
  margin: 0;
  padding-bottom: 0;
}

.comment {
  background: yellow;
}

.footnote, .footnoteref {
  font-size: 0.8em;
}

span.footnote, span.footnoteref {
  vertical-align: super;
}

#footnotes {
  margin: 20px 0 20px 0;
  padding: 7px 0 0 0;
}

#footnotes div.footnote {
  margin: 0 0 5px 0;
}

#footnotes hr {
  border: none;
  border-top: 1px solid silver;
  height: 1px;
  text-align: left;
  margin-left: 0;
  width: 20%;
  min-width: 100px;
}

div.colist td {
  padding-right: 0.5em;
  padding-bottom: 0.3em;
  vertical-align: top;
}
div.colist td img {
  margin-top: 0.3em;
}

@media print {
  #footer-badges { display: none; }
}

#toc {
  margin-bottom: 2.5em;
}

#toctitle {
  color: #527bbd;
  font-size: 1.1em;
  font-weight: bold;
  margin-top: 1.0em;
  margin-bottom: 0.1em;
}

div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
  margin-top: 0;
  margin-bottom: 0;
}
div.toclevel2 {
  margin-left: 2em;
  font-size: 0.9em;
}
div.toclevel3 {
  margin-left: 4em;
  font-size: 0.9em;
}
div.toclevel4 {
  margin-left: 6em;
  font-size: 0.9em;
}

span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }

span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }

span.big { font-size: 2em; }
span.small { font-size: 0.6em; }

span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }

div.unbreakable { page-break-inside: avoid; }

/*
 * xhtml11 specific
 *
 * */

div.tableblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
div.tableblock > table {
  border: 3px solid #527bbd;
}
thead, p.table.header {
  font-weight: bold;
  color: #527bbd;
}
p.table {
  margin-top: 0;
}
/* Because the table frame attribute is overridden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
  border-style: none;
}
div.tableblock > table[frame="hsides"] {
  border-left-style: none;
  border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
  border-top-style: none;
  border-bottom-style: none;
}

/*
 * html5 specific
 *
 * */

table.tableblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
thead, p.tableblock.header {
  font-weight: bold;
  color: #527bbd;
}
p.tableblock {
  margin-top: 0;
}
table.tableblock {
  border-width: 3px;
  border-spacing: 0px;
  border-style: solid;
  border-color: #527bbd;
  border-collapse: collapse;
}
th.tableblock, td.tableblock {
  border-width: 1px;
  padding: 4px;
  border-style: solid;
  border-color: #527bbd;
}

table.tableblock.frame-topbot {
  border-left-style: hidden;
  border-right-style: hidden;
}
table.tableblock.frame-sides {
  border-top-style: hidden;
  border-bottom-style: hidden;
}
table.tableblock.frame-none {
  border-style: hidden;
}

th.tableblock.halign-left, td.tableblock.halign-left {
  text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
  text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
  text-align: right;
}

th.tableblock.valign-top, td.tableblock.valign-top {
  vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
  vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
  vertical-align: bottom;
}

/*
 * manpage specific
 *
 * */

body.manpage h1 {
  padding-top: 0.5em;
  padding-bottom: 0.5em;
  border-top: 2px solid silver;
  border-bottom: 2px solid silver;
}
body.manpage h2 {
  border-style: none;
}
body.manpage div.sectionbody {
  margin-left: 3em;
}

@media print {
  body.manpage div#toc { display: none; }
}

</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = {  // Namespace.

/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////

/* Author: Mihai Bazon, September 2002
 * http://students.infoiasi.ro/~mishoo
 *
 * Table Of Content generator
 * Version: 0.4
 *
 * Feel free to use this script under the terms of the GNU General Public
 * License, as long as you do not remove or alter this notice.
 */

/* modified by Troy D. Hanson, September 2006. License: GPL */
 /* modified by Stuart Rackham, 2006, 2009. License: GPL */

// toclevels = 1..4.
toc: function (toclevels) {

function getText(el) {
    var text = "";
    for (var i = el.firstChild; i != null; i = i.nextSibling) {
      if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
        text += i.data;
      else if (i.firstChild != null)
        text += getText(i);
    }
    return text;
  }

function TocEntry(el, text, toclevel) {
    this.element = el;
    this.text = text;
    this.toclevel = toclevel;
  }

var toc = document.getElementById("toc");
  if (!toc) {
    return;
  }

// Delete existing TOC entries in case we're reloading the TOC.
  var tocEntriesToRemove = [];
  var i;
  for (i = 0; i < toc.childNodes.length; i++) {
    var entry = toc.childNodes[i];
    if (entry.nodeName.toLowerCase() == 'div'
     && entry.getAttribute("class")
     && entry.getAttribute("class").match(/^toclevel/))
      tocEntriesToRemove.push(entry);
  }
  for (i = 0; i < tocEntriesToRemove.length; i++) {
    toc.removeChild(tocEntriesToRemove[i]);
  }

// Rebuild TOC entries.
  var entries = tocEntries(document.getElementById("content"), toclevels);
  for (var i = 0; i < entries.length; ++i) {
    var entry = entries[i];
    if (entry.element.id == "")
      entry.element.id = "_toc_" + i;
    var a = document.createElement("a");
    a.href = "#" + entry.element.id;
    a.appendChild(document.createTextNode(entry.text));
    var div = document.createElement("div");
    div.appendChild(a);
    div.className = "toclevel" + entry.toclevel;
    toc.appendChild(div);
  }
  if (entries.length == 0)
    toc.parentNode.removeChild(toc);
},

/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////

/* Based on footnote generation code from:
 * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
 */

footnotes: function () {
  // Delete existing footnote entries in case we're reloading the footnodes.
  var i;
  var noteholder = document.getElementById("footnotes");
  if (!noteholder) {
    return;
  }
  var entriesToRemove = [];
  for (i = 0; i < noteholder.childNodes.length; i++) {
    var entry = noteholder.childNodes[i];
    if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
      entriesToRemove.push(entry);
  }
  for (i = 0; i < entriesToRemove.length; i++) {
    noteholder.removeChild(entriesToRemove[i]);
  }

// Rebuild footnote entries.
  var cont = document.getElementById("content");
  var spans = cont.getElementsByTagName("span");
  var refs = {};
  var n = 0;
  for (i=0; i<spans.length; i++) {
    if (spans[i].className == "footnote") {
      n++;
      var note = spans[i].getAttribute("data-note");
      if (!note) {
        // Use [\s\S] in place of . so multi-line matches work.
        // Because JavaScript has no s (dotall) regex flag.
        note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
        spans[i].innerHTML =
          "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
          "' title='View footnote' class='footnote'>" + n + "</a>]";
        spans[i].setAttribute("data-note", note);
      }
      noteholder.innerHTML +=
        "<div class='footnote' id='_footnote_" + n + "'>" +
        "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
        n + "</a>. " + note + "</div>";
      var id =spans[i].getAttribute("id");
      if (id != null) refs["#"+id] = n;
    }
  }
  if (n == 0)
    noteholder.parentNode.removeChild(noteholder);
  else {
    // Process footnoterefs.
    for (i=0; i<spans.length; i++) {
      if (spans[i].className == "footnoteref") {
        var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
        href = href.match(/#.*/)[0];  // Because IE return full URL.
        n = refs[href];
        spans[i].innerHTML =
          "[<a href='#_footnote_" + n +
          "' title='View footnote' class='footnote'>" + n + "</a>]";
      }
    }
  }
},

install: function(toclevels) {
  var timerId;

function reinstall() {
    asciidoc.footnotes();
    if (toclevels) {
      asciidoc.toc(toclevels);
    }
  }

function reinstallAndRemoveTimer() {
    clearInterval(timerId);
    reinstall();
  }

timerId = setInterval(reinstall, 500);
  if (document.addEventListener)
    document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
  else
    window.onload = reinstallAndRemoveTimer;
}

}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="manpage">
<div id="header">
<h1>
TRACE-CMD-PROFILE(1) Manual Page
</h1>
<h2>NAME</h2>
<div class="sectionbody">
<p>trace-cmd-profile -
   profile tasks running live
</p>
</div>
</div>
<div id="content">
<div class="sect1">
<h2 id="_synopsis">SYNOPSIS</h2>
<div class="sectionbody">
<div class="paragraph"><p><strong>trace-cmd profile [<em>OPTIONS</em>]</strong> [<em>command</em>]</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_description">DESCRIPTION</h2>
<div class="sectionbody">
<div class="paragraph"><p>The trace-cmd(1) profile will start tracing just like trace-cmd-record(1),
with the <strong>--profile</strong> option, except that it does not write to a file,
but instead, it will read the events as they happen and will update the accounting
of the events. When the trace is finished, it will report the results just like
trace-cmd-report(1) would do with its <strong>--profile</strong> option. In other words,
the profile command does the work of trace-cmd record --profile, and trace-cmd
report --profile without having to record the data to disk, in between.</p></div>
<div class="paragraph"><p>The advantage of using the profile command is that the profiling can be done
over a long period of time where recording all events would take up too much
disk space.</p></div>
<div class="paragraph"><p>This will enable several events as well as the function graph tracer
with a depth of one (if the kernel supports it). This is to show where
tasks enter and exit the kernel and how long they were in the kernel.</p></div>
<div class="paragraph"><p>To disable calling function graph, use the <strong>-p</strong> option to enable another
tracer. To not enable any tracer, use <strong>-p nop</strong>.</p></div>
<div class="paragraph"><p>All timings are currently in nanoseconds.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_options">OPTIONS</h2>
<div class="sectionbody">
<div class="paragraph"><p>These are the same as trace-cmd-record(1) with the <strong>--profile</strong> option.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<strong>-p</strong> <em>tracer</em>
</dt>
<dd>
<p>
    Set a tracer plugin to run instead of function graph tracing set to
    depth of 1. To not run any tracer, use <strong>-p nop</strong>.
</p>
</dd>
<dt class="hdlist1">
<strong>-S</strong>
</dt>
<dd>
<p>
    Only enable the tracer or events speficied on the command line.
    With this option, the function_graph tracer is not enabled, nor are
    any events (like sched_switch), unless they are specifically specified
    on the command line (i.e. -p function -e sched_switch -e sched_wakeup)
</p>
</dd>
<dt class="hdlist1">
<strong>-G</strong>
</dt>
<dd>
<p>
    Set interrupt (soft and hard) events as global (associated to CPU
    instead of tasks).
</p>
</dd>
<dt class="hdlist1">
<strong>-o</strong> <em>file</em>
</dt>
<dd>
<p>
    Write the output of the profile to <em>file</em>. This supersedes <strong>--stderr</strong>
</p>
</dd>
<dt class="hdlist1">
<strong>-H</strong> <em>event-hooks</em>
</dt>
<dd>
<p>
    Add custom event matching to connect any two events together. Format is:
    [&lt;start_system&gt;:]&lt;start_event&gt;,&lt;start_match&gt;[,&lt;start_pid&gt;]/
    [&lt;end_system&gt;:]&lt;end_event&gt;,&lt;end_match&gt;[,&lt;flags&gt;]
</p>
<div class="literalblock">
<div class="content">
<pre><code>The start_system:start_event (start_system is optional), is the event that
starts the timing.</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>start_match is the field in the start event that is to match with the
end_match in the end event.</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>start_pid is optional, as matches are attached to the tasks that run
the events, if another field should be used to find that task, then
it is specified with start_pid.</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>end_system:end_event is the event that ends the timing (end_system is
optional).</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>end_match is the field in end_match that wil match the start event field
start_match.</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>flags are optional and can be the following (case insensitive):</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>p : The two events are pinned to the same CPU (start and end happen
    on the same CPU always).</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>s : The event should have a stack traced with it (enable stack tracing
    for the start event).</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>g : The event is global (not associated to a task). start_pid is
    not applicable with this flag.</code></pre>
</div></div>
</dd>
<dt class="hdlist1">
<strong>--stderr</strong>
</dt>
<dd>
<p>
    Redirect the output to stderr. The output of the command being executed
    is not changed. This allows watching the command execute and saving the
    output of the profile to another file.
</p>
</dd>
<dt class="hdlist1">
<strong>--verbose</strong>[=<em>level</em>]
</dt>
<dd>
<p>
     Set the log level. Supported log levels are "none", "critical", "error", "warning",
     "info", "debug", "all" or their identifiers "0", "1", "2", "3", "4", "5", "6". Setting the log
     level to specific value enables all logs from that and all previous levels.
     The level will default to "info" if one is not specified.
</p>
<div class="literalblock">
<div class="content">
<pre><code>Example: enable all critical, error and warning logs</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>trace-cmd profile --verbose=warning</code></pre>
</div></div>
</dd>
</dl></div>
</div>
</div>
<div class="sect1">
<h2 id="_examples">EXAMPLES</h2>
<div class="sectionbody">
<div class="literalblock">
<div class="content">
<pre><code> ---
# trace-cmd profile -F sleep 1
 [..]
task: sleep-1121
  Event: sched_switch:R (2) Total: 234559 Avg: 117279 Max: 129886 Min:104673
          |
          + ftrace_raw_event_sched_switch (0xffffffff8109f310)
              100% (2) time:234559 max:129886 min:104673 avg:117279
               __schedule (0xffffffff816c1e81)
               preempt_schedule (0xffffffff816c236e)
               ___preempt_schedule (0xffffffff81351a59)
                |
                + unmap_single_vma (0xffffffff81198c05)
                |   55% (1) time:129886 max:129886 min:0 avg:129886
                |    stop_one_cpu (0xffffffff8110909a)
                |    sched_exec (0xffffffff810a119b)
                |    do_execveat_common.isra.31 (0xffffffff811de528)
                |    do_execve (0xffffffff811dea8c)
                |    SyS_execve (0xffffffff811ded1e)
                |    return_to_handler (0xffffffff816c8458)
                |    stub_execve (0xffffffff816c6929)
                |
                + unmap_single_vma (0xffffffff81198c05)
                    45% (1) time:104673 max:104673 min:0 avg:104673
                     unmap_vmas (0xffffffff81199174)
                     exit_mmap (0xffffffff811a1f5b)
                     mmput (0xffffffff8107699a)
                     flush_old_exec (0xffffffff811ddb75)
                     load_elf_binary (0xffffffff812287df)
                     search_binary_handler (0xffffffff811dd3e0)
                     do_execveat_common.isra.31 (0xffffffff811de8bd)
                     do_execve (0xffffffff811dea8c)
                     SyS_execve (0xffffffff811ded1e)
                     return_to_handler (0xffffffff816c8458)
                     stub_execve (0xffffffff816c6929)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: sched_switch:S (1) Total: 1000513242 Avg: 1000513242 Max: 1000513242 Min:1000513242
        |
        + ftrace_raw_event_sched_switch (0xffffffff8109f310)
            100% (1) time:1000513242 max:1000513242 min:0 avg:1000513242
             __schedule (0xffffffff816c1e81)
             schedule (0xffffffff816c23b9)
             do_nanosleep (0xffffffff816c4f1c)
             hrtimer_nanosleep (0xffffffff810dcd86)
             SyS_nanosleep (0xffffffff810dcea6)
             return_to_handler (0xffffffff816c8458)
             tracesys_phase2 (0xffffffff816c65b0)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: sched_wakeup:1121 (1) Total: 43405 Avg: 43405 Max: 43405 Min:43405
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (1) time:43405 max:43405 min:0 avg:43405
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             wake_up_process (0xffffffff810a4057)
             hrtimer_wakeup (0xffffffff810db772)
             __run_hrtimer (0xffffffff810dbd91)
             hrtimer_interrupt (0xffffffff810dc6b7)
             local_apic_timer_interrupt (0xffffffff810363e7)
             smp_trace_apic_timer_interrupt (0xffffffff816c8c6a)
             trace_apic_timer_interrupt (0xffffffff816c725a)
             finish_task_switch (0xffffffff8109c3a4)
             __schedule (0xffffffff816c1e01)
             schedule (0xffffffff816c23b9)
             ring_buffer_wait (0xffffffff811323a3)
             wait_on_pipe (0xffffffff81133d93)
             tracing_buffers_splice_read (0xffffffff811350b0)
             do_splice_to (0xffffffff8120476f)
             SyS_splice (0xffffffff81206c1f)
             tracesys_phase2 (0xffffffff816c65b0)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: func: sys_nanosleep() (1) Total: 1000598016 Avg: 1000598016 Max: 1000598016 Min:1000598016
Event: func: sys_munmap() (1) Total: 14300 Avg: 14300 Max: 14300 Min:14300
Event: func: sys_arch_prctl() (1) Total: 571 Avg: 571 Max: 571 Min:571
Event: func: sys_mprotect() (4) Total: 14382 Avg: 3595 Max: 7196 Min:2190
Event: func: SyS_read() (1) Total: 2640 Avg: 2640 Max: 2640 Min:2640
Event: func: sys_close() (5) Total: 4001 Avg: 800 Max: 1252 Min:414
Event: func: sys_newfstat() (3) Total: 11684 Avg: 3894 Max: 10206 Min:636
Event: func: SyS_open() (3) Total: 23615 Avg: 7871 Max: 10535 Min:4743
Event: func: sys_access() (1) Total: 5924 Avg: 5924 Max: 5924 Min:5924
Event: func: SyS_mmap() (8) Total: 39153 Avg: 4894 Max: 12354 Min:1518
Event: func: smp_trace_apic_timer_interrupt() (1) Total: 10298 Avg: 10298 Max: 10298 Min:10298
Event: func: SyS_brk() (4) Total: 2407 Avg: 601 Max: 1564 Min:206
Event: func: do_notify_resume() (2) Total: 4095 Avg: 2047 Max: 2521 Min:1574
Event: func: sys_execve() (5) Total: 1625251 Avg: 325050 Max: 1605698 Min:3570
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (1) time:1605698 max:1605698 min:0 avg:1605698
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             wake_up_process (0xffffffff810a4057)
             cpu_stop_queue_work (0xffffffff81108df8)
             stop_one_cpu (0xffffffff8110909a)
             sched_exec (0xffffffff810a119b)
             do_execveat_common.isra.31 (0xffffffff811de528)
             do_execve (0xffffffff811dea8c)
             SyS_execve (0xffffffff811ded1e)
             return_to_handler (0xffffffff816c8458)
             stub_execve (0xffffffff816c6929)
             stub_execve (0xffffffff816c6929)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: func: syscall_trace_enter_phase2() (38) Total: 21544 Avg: 566 Max: 1066 Min:329
Event: func: syscall_trace_enter_phase1() (38) Total: 9202 Avg: 242 Max: 376 Min:150
Event: func: __do_page_fault() (53) Total: 257672 Avg: 4861 Max: 27745 Min:458
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (1) time:27745 max:27745 min:0 avg:27745
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             default_wake_function (0xffffffff810a4002)
             autoremove_wake_function (0xffffffff810b50fd)
             __wake_up_common (0xffffffff810b4958)
             __wake_up (0xffffffff810b4cb8)
             rb_wake_up_waiters (0xffffffff8112f126)
             irq_work_run_list (0xffffffff81157d0f)
             irq_work_run (0xffffffff81157d5e)
             smp_trace_irq_work_interrupt (0xffffffff810082fc)
             trace_irq_work_interrupt (0xffffffff816c7aaa)
             return_to_handler (0xffffffff816c8458)
             trace_do_page_fault (0xffffffff810478b2)
             trace_page_fault (0xffffffff816c7dd2)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: func: syscall_trace_leave() (38) Total: 26145 Avg: 688 Max: 1264 Min:381
Event: func: __sb_end_write() (1) Total: 373 Avg: 373 Max: 373 Min:373
Event: func: fsnotify() (1) Total: 598 Avg: 598 Max: 598 Min:598
Event: func: __fsnotify_parent() (1) Total: 286 Avg: 286 Max: 286 Min:286
Event: func: mutex_unlock() (2) Total: 39636 Avg: 19818 Max: 39413 Min:223
Event: func: smp_trace_irq_work_interrupt() (6) Total: 236459 Avg: 39409 Max: 100671 Min:634
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (4) time:234348 max:100671 min:38745 avg:58587
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             default_wake_function (0xffffffff810a4002)
             autoremove_wake_function (0xffffffff810b50fd)
             __wake_up_common (0xffffffff810b4958)
             __wake_up (0xffffffff810b4cb8)
             rb_wake_up_waiters (0xffffffff8112f126)
             irq_work_run_list (0xffffffff81157d0f)
             irq_work_run (0xffffffff81157d5e)
             smp_trace_irq_work_interrupt (0xffffffff810082fc)
             return_to_handler (0xffffffff816c8458)
             trace_irq_work_interrupt (0xffffffff816c7aaa)
              |
              + ftrace_return_to_handler (0xffffffff81140840)
              |   84% (3) time:197396 max:100671 min:38745 avg:65798
              |    return_to_handler (0xffffffff816c846d)
              |    trace_page_fault (0xffffffff816c7dd2)
              |
              + ftrace_return_to_handler (0xffffffff81140840)
                  16% (1) time:36952 max:36952 min:0 avg:36952
                   ftrace_graph_caller (0xffffffff816c8428)
                   mutex_unlock (0xffffffff816c3f75)
                   rb_simple_write (0xffffffff81133142)
                   vfs_write (0xffffffff811d7727)
                   SyS_write (0xffffffff811d7acf)
                   tracesys_phase2 (0xffffffff816c65b0)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: sys_enter:35 (1) Total: 1000599765 Avg: 1000599765 Max: 1000599765 Min:1000599765
Event: sys_enter:11 (1) Total: 55025 Avg: 55025 Max: 55025 Min:55025
Event: sys_enter:158 (1) Total: 1584 Avg: 1584 Max: 1584 Min:1584
Event: sys_enter:10 (4) Total: 18359 Avg: 4589 Max: 8764 Min:2933
Event: sys_enter:0 (1) Total: 4223 Avg: 4223 Max: 4223 Min:4223
Event: sys_enter:3 (5) Total: 9948 Avg: 1989 Max: 2606 Min:1203
Event: sys_enter:5 (3) Total: 15530 Avg: 5176 Max: 11840 Min:1405
Event: sys_enter:2 (3) Total: 28002 Avg: 9334 Max: 12035 Min:5656
Event: sys_enter:21 (1) Total: 7814 Avg: 7814 Max: 7814 Min:7814
Event: sys_enter:9 (8) Total: 49583 Avg: 6197 Max: 14137 Min:2362
Event: sys_enter:12 (4) Total: 108493 Avg: 27123 Max: 104079 Min:922
Event: sys_enter:59 (5) Total: 1631608 Avg: 326321 Max: 1607529 Min:4563
Event: page_fault_user:0x398d86b630 (1)
Event: page_fault_user:0x398d844de0 (1)
Event: page_fault_user:0x398d8d9020 (1)
Event: page_fault_user:0x1d37008 (1)
Event: page_fault_user:0x7f0b89e91074 (1)
Event: page_fault_user:0x7f0b89d98ed0 (1)
Event: page_fault_user:0x7f0b89ec8950 (1)
Event: page_fault_user:0x7f0b89d83644 (1)
Event: page_fault_user:0x7f0b89d622a8 (1)
Event: page_fault_user:0x7f0b89d5a560 (1)
Event: page_fault_user:0x7f0b89d34010 (1)
Event: page_fault_user:0x1d36008 (1)
Event: page_fault_user:0x398d900510 (1)
Event: page_fault_user:0x398dbb3ae8 (1)
Event: page_fault_user:0x398d87f490 (1)
Event: page_fault_user:0x398d8eb660 (1)
Event: page_fault_user:0x398d8bd730 (1)
Event: page_fault_user:0x398d9625d9 (1)
Event: page_fault_user:0x398d931810 (1)
Event: page_fault_user:0x398dbb7114 (1)
Event: page_fault_user:0x398d837610 (1)
Event: page_fault_user:0x398d89e860 (1)
Event: page_fault_user:0x398d8f23b0 (1)
Event: page_fault_user:0x398dbb4510 (1)
Event: page_fault_user:0x398dbad6f0 (1)
Event: page_fault_user:0x398dbb1018 (1)
Event: page_fault_user:0x398d977b37 (1)
Event: page_fault_user:0x398d92eb60 (1)
Event: page_fault_user:0x398d8abff0 (1)
Event: page_fault_user:0x398dbb0d30 (1)
Event: page_fault_user:0x398dbb6c24 (1)
Event: page_fault_user:0x398d821c50 (1)
Event: page_fault_user:0x398dbb6c20 (1)
Event: page_fault_user:0x398d886350 (1)
Event: page_fault_user:0x7f0b90125000 (1)
Event: page_fault_user:0x7f0b90124740 (1)
Event: page_fault_user:0x7f0b90126000 (1)
Event: page_fault_user:0x398d816230 (1)
Event: page_fault_user:0x398d8002b8 (1)
Event: page_fault_user:0x398dbb0b40 (1)
Event: page_fault_user:0x398dbb2880 (1)
Event: page_fault_user:0x7f0b90141cc6 (1)
Event: page_fault_user:0x7f0b9013b85c (1)
Event: page_fault_user:0x7f0b90127000 (1)
Event: page_fault_user:0x606e70 (1)
Event: page_fault_user:0x7f0b90144010 (1)
Event: page_fault_user:0x7fffcb31b038 (1)
Event: page_fault_user:0x606da8 (1)
Event: page_fault_user:0x400040 (1)
Event: page_fault_user:0x398d222218 (1)
Event: page_fault_user:0x398d015120 (1)
Event: page_fault_user:0x398d220ce8 (1)
Event: page_fault_user:0x398d220b80 (1)
Event: page_fault_user:0x7fffcb2fcff8 (1)
Event: page_fault_user:0x398d001590 (1)
Event: page_fault_user:0x398d838490 (1)
Event: softirq_raise:RCU (3) Total: 252931 Avg: 84310 Max: 243288 Min:4639
Event: softirq_raise:SCHED (2) Total: 241249 Avg: 120624 Max: 239076 Min:2173
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (1) time:239076 max:239076 min:0 avg:239076
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             default_wake_function (0xffffffff810a4002)
             autoremove_wake_function (0xffffffff810b50fd)
             __wake_up_common (0xffffffff810b4958)
             __wake_up (0xffffffff810b4cb8)
             rb_wake_up_waiters (0xffffffff8112f126)
             irq_work_run_list (0xffffffff81157d0f)
             irq_work_run (0xffffffff81157d5e)
             smp_trace_irq_work_interrupt (0xffffffff810082fc)
             trace_irq_work_interrupt (0xffffffff816c7aaa)
             irq_exit (0xffffffff8107dd66)
             smp_trace_apic_timer_interrupt (0xffffffff816c8c7a)
             trace_apic_timer_interrupt (0xffffffff816c725a)
             prepare_ftrace_return (0xffffffff8103d4fd)
             ftrace_graph_caller (0xffffffff816c8428)
             mem_cgroup_begin_page_stat (0xffffffff811cfd25)
             page_remove_rmap (0xffffffff811a4fc5)
             stub_execve (0xffffffff816c6929)
             unmap_single_vma (0xffffffff81198b1c)
             unmap_vmas (0xffffffff81199174)
             exit_mmap (0xffffffff811a1f5b)
             mmput (0xffffffff8107699a)
             flush_old_exec (0xffffffff811ddb75)
             load_elf_binary (0xffffffff812287df)
             search_binary_handler (0xffffffff811dd3e0)
             do_execveat_common.isra.31 (0xffffffff811de8bd)
             do_execve (0xffffffff811dea8c)
             SyS_execve (0xffffffff811ded1e)
             return_to_handler (0xffffffff816c8458)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: softirq_raise:HI (3) Total: 72472 Avg: 24157 Max: 64186 Min:3430
Event: softirq_entry:RCU (2) Total: 3191 Avg: 1595 Max: 1788 Min:1403
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (1) time:1788 max:1788 min:0 avg:1788
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             default_wake_function (0xffffffff810a4002)
             autoremove_wake_function (0xffffffff810b50fd)
             __wake_up_common (0xffffffff810b4958)
             __wake_up (0xffffffff810b4cb8)
             rb_wake_up_waiters (0xffffffff8112f126)
             irq_work_run_list (0xffffffff81157d0f)
             irq_work_run (0xffffffff81157d5e)
             smp_trace_irq_work_interrupt (0xffffffff810082fc)
             trace_irq_work_interrupt (0xffffffff816c7aaa)
             irq_work_queue (0xffffffff81157e95)
             ring_buffer_unlock_commit (0xffffffff8113039f)
             __buffer_unlock_commit (0xffffffff811367d5)
             trace_buffer_unlock_commit (0xffffffff811376a2)
             ftrace_event_buffer_commit (0xffffffff81146d5f)
             ftrace_raw_event_sched_process_exec (0xffffffff8109c511)
             do_execveat_common.isra.31 (0xffffffff811de9a3)
             do_execve (0xffffffff811dea8c)
             SyS_execve (0xffffffff811ded1e)
             return_to_handler (0xffffffff816c8458)
             stub_execve (0xffffffff816c6929)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code> Event: softirq_entry:SCHED (2) Total: 2289 Avg: 1144 Max: 1350 Min:939
 Event: softirq_entry:HI (3) Total: 180146 Avg: 60048 Max: 178969 Min:499
         |
         + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
             100% (1) time:178969 max:178969 min:0 avg:178969
              ttwu_do_wakeup (0xffffffff810a01a2)
              ttwu_do_activate.constprop.122 (0xffffffff810a0236)
              try_to_wake_up (0xffffffff810a3ec3)
              wake_up_process (0xffffffff810a4057)
              wake_up_worker (0xffffffff8108de74)
              insert_work (0xffffffff8108fca6)
              __queue_work (0xffffffff8108fe12)
              delayed_work_timer_fn (0xffffffff81090088)
              call_timer_fn (0xffffffff810d8f89)
              run_timer_softirq (0xffffffff810da8a1)
              __do_softirq (0xffffffff8107d8fa)
              irq_exit (0xffffffff8107dd66)
              smp_trace_apic_timer_interrupt (0xffffffff816c8c7a)
              trace_apic_timer_interrupt (0xffffffff816c725a)
              prepare_ftrace_return (0xffffffff8103d4fd)
              ftrace_graph_caller (0xffffffff816c8428)
              mem_cgroup_begin_page_stat (0xffffffff811cfd25)
              page_remove_rmap (0xffffffff811a4fc5)
              stub_execve (0xffffffff816c6929)
              unmap_single_vma (0xffffffff81198b1c)
              unmap_vmas (0xffffffff81199174)
              exit_mmap (0xffffffff811a1f5b)
              mmput (0xffffffff8107699a)
              flush_old_exec (0xffffffff811ddb75)
              load_elf_binary (0xffffffff812287df)
              search_binary_handler (0xffffffff811dd3e0)
              do_execveat_common.isra.31 (0xffffffff811de8bd)
              do_execve (0xffffffff811dea8c)
              SyS_execve (0xffffffff811ded1e)
              return_to_handler (0xffffffff816c8458)
---</code></pre>
</div></div>
<div class="paragraph"><p>The above uses <strong>-F</strong> to follow the sleep task. It filters only on events
that pertain to sleep. Note, in order to follow forks, you need to also
include the <strong>-c</strong> flag.</p></div>
<div class="paragraph"><p>Other tasks will appear in the profile as well if events reference more
than one task (like sched_switch and sched_wakeup do. The "prev_pid" and
"next_pid" of sched_switch, and the "common_pid" and "pid" of sched_wakeup).</p></div>
<div class="paragraph"><p>Stack traces are attached to events that are related to them.</p></div>
<div class="paragraph"><p>Taking a look at the above output:</p></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: sched_switch:R (2) Total: 234559 Avg: 117279 Max: 129886 Min:104673</code></pre>
</div></div>
<div class="paragraph"><p>This shows that task was preempted (it&#8217;s in the running <em>R</em> state).
It was preempted twice <em>(2)</em> for a total of 234,559 nanoseconds, with a average
preempt time of 117,279 ns, and maximum of 128,886 ns and minimum of 104,673 ns.</p></div>
<div class="paragraph"><p>The tree shows where it was preempted:</p></div>
<div class="literalblock">
<div class="content">
<pre><code>|
+ ftrace_raw_event_sched_switch (0xffffffff8109f310)
    100% (2) time:234559 max:129886 min:104673 avg:117279
     __schedule (0xffffffff816c1e81)
     preempt_schedule (0xffffffff816c236e)
     ___preempt_schedule (0xffffffff81351a59)
      |
      + unmap_single_vma (0xffffffff81198c05)
      |   55% (1) time:129886 max:129886 min:0 avg:129886
      |    stop_one_cpu (0xffffffff8110909a)
      |    sched_exec (0xffffffff810a119b)
      |    do_execveat_common.isra.31 (0xffffffff811de528)
      |    do_execve (0xffffffff811dea8c)
      |    SyS_execve (0xffffffff811ded1e)
      |    return_to_handler (0xffffffff816c8458)
      |    stub_execve (0xffffffff816c6929)
      |
      + unmap_single_vma (0xffffffff81198c05)
          45% (1) time:104673 max:104673 min:0 avg:104673
           unmap_vmas (0xffffffff81199174)
           exit_mmap (0xffffffff811a1f5b)
           mmput (0xffffffff8107699a)
           flush_old_exec (0xffffffff811ddb75)
           load_elf_binary (0xffffffff812287df)
           search_binary_handler (0xffffffff811dd3e0)
           do_execveat_common.isra.31 (0xffffffff811de8bd)
           do_execve (0xffffffff811dea8c)
           SyS_execve (0xffffffff811ded1e)
           return_to_handler (0xffffffff816c8458)
           stub_execve (0xffffffff816c6929)</code></pre>
</div></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: sched_switch:S (1) Total: 1000513242 Avg: 1000513242 Max: 1000513242 Min:10005132</code></pre>
</div></div>
<div class="paragraph"><p>This shows that the task was scheduled out in the INTERRUPTIBLE state once
for a total of 1,000,513,242 ns (~1s), which makes sense as the task was a
"sleep 1".</p></div>
<div class="paragraph"><p>After the schedule events, the function events are shown. By default the
profiler will use the function graph tracer if the depth setting is supported
by the kernel. It will set the depth to one which will only trace the first
function that enters the kernel. It will also record the amount of time
it was in the kernel.</p></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: func: sys_nanosleep() (1) Total: 1000598016 Avg: 1000598016 Max: 1000598016 Min:1000598016
Event: func: sys_munmap() (1) Total: 14300 Avg: 14300 Max: 14300 Min:14300
Event: func: sys_arch_prctl() (1) Total: 571 Avg: 571 Max: 571 Min:571
Event: func: sys_mprotect() (4) Total: 14382 Avg: 3595 Max: 7196 Min:2190
Event: func: SyS_read() (1) Total: 2640 Avg: 2640 Max: 2640 Min:2640
Event: func: sys_close() (5) Total: 4001 Avg: 800 Max: 1252 Min:414
Event: func: sys_newfstat() (3) Total: 11684 Avg: 3894 Max: 10206 Min:636
Event: func: SyS_open() (3) Total: 23615 Avg: 7871 Max: 10535 Min:4743
Event: func: sys_access() (1) Total: 5924 Avg: 5924 Max: 5924 Min:5924
Event: func: SyS_mmap() (8) Total: 39153 Avg: 4894 Max: 12354 Min:1518
Event: func: smp_trace_apic_timer_interrupt() (1) Total: 10298 Avg: 10298 Max: 10298 Min:10298
Event: func: SyS_brk() (4) Total: 2407 Avg: 601 Max: 1564 Min:206
Event: func: do_notify_resume() (2) Total: 4095 Avg: 2047 Max: 2521 Min:1574
Event: func: sys_execve() (5) Total: 1625251 Avg: 325050 Max: 1605698 Min:3570</code></pre>
</div></div>
<div class="paragraph"><p>Count of times the event was hit is always in parenthesis <em>(5)</em>.</p></div>
<div class="paragraph"><p>The function graph trace may produce too much overhead as it is still
triggering (just not tracing) on all functions. To limit functions just to
system calls (not interrupts), add the following option:</p></div>
<div class="literalblock">
<div class="content">
<pre><code>-l 'sys_*' -l 'SyS_*'</code></pre>
</div></div>
<div class="paragraph"><p>To disable function graph tracing totally, use:</p></div>
<div class="literalblock">
<div class="content">
<pre><code>-p nop</code></pre>
</div></div>
<div class="paragraph"><p>To use function tracing instead (note, this will not record timings, but just
the count of times a function is hit):</p></div>
<div class="literalblock">
<div class="content">
<pre><code>-p function</code></pre>
</div></div>
<div class="paragraph"><p>Following the functions are the events that are recorded.</p></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: sys_enter:35 (1) Total: 1000599765 Avg: 1000599765 Max: 1000599765 Min:1000599765
Event: sys_enter:11 (1) Total: 55025 Avg: 55025 Max: 55025 Min:55025
Event: sys_enter:158 (1) Total: 1584 Avg: 1584 Max: 1584 Min:1584
Event: sys_enter:10 (4) Total: 18359 Avg: 4589 Max: 8764 Min:2933
Event: sys_enter:0 (1) Total: 4223 Avg: 4223 Max: 4223 Min:4223
Event: sys_enter:3 (5) Total: 9948 Avg: 1989 Max: 2606 Min:1203
Event: sys_enter:5 (3) Total: 15530 Avg: 5176 Max: 11840 Min:1405
Event: sys_enter:2 (3) Total: 28002 Avg: 9334 Max: 12035 Min:5656
Event: sys_enter:21 (1) Total: 7814 Avg: 7814 Max: 7814 Min:7814
Event: sys_enter:9 (8) Total: 49583 Avg: 6197 Max: 14137 Min:2362
Event: sys_enter:12 (4) Total: 108493 Avg: 27123 Max: 104079 Min:922
Event: sys_enter:59 (5) Total: 1631608 Avg: 326321 Max: 1607529 Min:4563</code></pre>
</div></div>
<div class="paragraph"><p>These are the raw system call events, with the raw system call ID after
the "sys_enter:"  For example, "59" is execve(2). Why did it execute 5 times?
Looking at a strace of this run, we can see:</p></div>
<div class="literalblock">
<div class="content">
<pre><code>execve("/usr/lib64/ccache/sleep", ["sleep", "1"], [/* 27 vars */] &lt;unfinished ...&gt;
&lt;... execve resumed&gt; )      = -1 ENOENT (No such file or directory)
execve("/usr/local/sbin/sleep", ["sleep", "1"], [/* 27 vars */] &lt;unfinished ...&gt;
&lt;... execve resumed&gt; )      = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/sleep", ["sleep", "1"], [/* 27 vars */] &lt;unfinished ...&gt;
&lt;... execve resumed&gt; )      = -1 ENOENT (No such file or directory)
execve("/usr/sbin/sleep", ["sleep", "1"], [/* 27 vars */] &lt;unfinished ...&gt;
&lt;... execve resumed&gt; )      = -1 ENOENT (No such file or directory)
execve("/usr/bin/sleep", ["sleep", "1"], [/* 27 vars */] &lt;unfinished ...&gt;
&lt;... execve resumed&gt; )      = 0</code></pre>
</div></div>
<div class="paragraph"><p>It attempted to execve the "sleep" command for each path in $PATH until it found
one.</p></div>
<div class="paragraph"><p>The page_fault_user events show what userspace address took a page fault.</p></div>
<div class="literalblock">
<div class="content">
<pre><code>Event: softirq_raise:RCU (3) Total: 252931 Avg: 84310 Max: 243288 Min:4639
Event: softirq_raise:SCHED (2) Total: 241249 Avg: 120624 Max: 239076 Min:2173
        |
        + ftrace_raw_event_sched_wakeup_template (0xffffffff8109d960)
            100% (1) time:239076 max:239076 min:0 avg:239076
             ttwu_do_wakeup (0xffffffff810a01a2)
             ttwu_do_activate.constprop.122 (0xffffffff810a0236)
             try_to_wake_up (0xffffffff810a3ec3)
             default_wake_function (0xffffffff810a4002)
             autoremove_wake_function (0xffffffff810b50fd)
             __wake_up_common (0xffffffff810b4958)
             __wake_up (0xffffffff810b4cb8)
             rb_wake_up_waiters (0xffffffff8112f126)
             irq_work_run_list (0xffffffff81157d0f)
             irq_work_run (0xffffffff81157d5e)
             smp_trace_irq_work_interrupt (0xffffffff810082fc)
             trace_irq_work_interrupt (0xffffffff816c7aaa)
             irq_exit (0xffffffff8107dd66)</code></pre>
</div></div>
<div class="paragraph"><p>The timings for the softirq_raise events measure the time it took from the raised
softirq to the time it executed.</p></div>
<div class="paragraph"><p>The timings for the softirq_entry events measure the time the softirq took to
execute.</p></div>
<div class="paragraph"><p>The stack traces for the softirqs (and possibly other events) are used when
an event has a stack attached to it. This can happen if the profile ran
more stacks than just the sched events, or when events are dropped and
stacks</p></div>
<div class="paragraph"><p>To have full control of what gets traced, use the <strong>-S</strong> option that will have
trace-cmd not enable any events or the function_graph tracer. Only the events
listed on the command line are shown.</p></div>
<div class="paragraph"><p>If only the time of kmalloc is needed to be seen, and where it was recorded,
using the <strong>-S</strong> option and enabling function_graph and stack tracing for just
the function needed will give the profile of only that function.</p></div>
<div class="literalblock">
<div class="content">
<pre><code> ---
# trace-cmd profile -S -p function_graph -l '*kmalloc*' -l '*kmalloc*:stacktrace' sleep 1
task: sshd-11786
  Event: func: __kmalloc_reserve.isra.59() (2) Total: 149684 Avg: 74842 Max: 75598 Min:74086
          |
          + __alloc_skb (0xffffffff815a8917)
          |   67% (2) time:149684 max:75598 min:74086 avg:74842
          |    __kmalloc_node_track_caller (0xffffffff811c6635)
          |    __kmalloc_reserve.isra.59 (0xffffffff815a84ac)
          |    return_to_handler (0xffffffff816c8458)
          |    sk_stream_alloc_skb (0xffffffff81604ea1)
          |    tcp_sendmsg (0xffffffff8160592c)
          |    inet_sendmsg (0xffffffff8162fed1)
          |    sock_aio_write (0xffffffff8159f9fc)
          |    do_sync_write (0xffffffff811d694a)
          |    vfs_write (0xffffffff811d7825)
          |    SyS_write (0xffffffff811d7adf)
          |    system_call_fastpath (0xffffffff816c63d2)
          |
          + __alloc_skb (0xffffffff815a8917)
              33% (1) time:74086 max:74086 min:74086 avg:74086
               __alloc_skb (0xffffffff815a8917)
               sk_stream_alloc_skb (0xffffffff81604ea1)
               tcp_sendmsg (0xffffffff8160592c)
               inet_sendmsg (0xffffffff8162fed1)
               sock_aio_write (0xffffffff8159f9fc)
               do_sync_write (0xffffffff811d694a)
               vfs_write (0xffffffff811d7825)
               SyS_write (0xffffffff811d7adf)
               system_call_fastpath (0xffffffff816c63d2)
 [..]
---</code></pre>
</div></div>
<div class="paragraph"><p>To watch the command run but save the output of the profile to a file
use --stderr, and redirect stderr to a file</p></div>
<div class="paragraph"><p># trace-cmd profile --stderr cyclictest -p 80 -n -t1 2&gt; profile.out</p></div>
<div class="paragraph"><p>Or simple use <strong>-o</strong></p></div>
<div class="paragraph"><p># trace-cmd profile -o profile.out cyclictest -p 80 -n -t1</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_see_also">SEE ALSO</h2>
<div class="sectionbody">
<div class="paragraph"><p>trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
trace-cmd-stop(1), trace-cmd-reset(1), trace-cmd-split(1),
trace-cmd-list(1), trace-cmd-listen(1)</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_author">AUTHOR</h2>
<div class="sectionbody">
<div class="paragraph"><p>Written by Steven Rostedt, &lt;<a href="mailto:rostedt@goodmis.org">rostedt@goodmis.org</a>&gt;</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_resources">RESOURCES</h2>
<div class="sectionbody">
<div class="paragraph"><p><a href="https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/">https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/</a></p></div>
</div>
</div>
<div class="sect1">
<h2 id="_copying">COPYING</h2>
<div class="sectionbody">
<div class="paragraph"><p>Copyright (C) 2014 Red Hat, Inc. Free use of this software is granted under
the terms of the GNU Public License (GPL).</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated
 2023-06-07 20:21:00 UTC
</div>
</div>
</body>
</html>

❌ Cancel

🗂️ Enhanced PHP File Manager

📊 Server Information

🔧 Quick Actions

✏️ Editing: trace-cmd-profile.1.html

📤 Upload File

📁 Create Folder

📄 Create File

✏️ Rename Item

🗑️ Delete Confirmation