[gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode?

Sven Oehme oehmes at gmail.com
Wed Mar 8 13:37:15 GMT 2017


given that i love data points, let me put some behind this thread :-)

starting in version 3.4 we enhanced the trace code of scale significant.
this went on release to release all the way up to 4.2.1. since 4.2.1 we
made further improvements, but much smaller changes, more optimization ,
e.g. reducing of trace levels verbosity, etc .
with 4.2.2  we switched from blocking traces to in memory traces as the
default trace infrastructure, this infrastructure was designed to be turned
on all the time with minimal impact on performance.
i just took a 4.2.3 build and did load it on one of my faster NVMe nodes
and did a 100% random read test with 64 threads against very fast storage.
while this was running i started trace with mmtracectl --start ; sleep 10 ;
mmtracectl --off , attached is a screenshot of this run :

[image: pasted1]
the trace started at 14:19:15 and stopped at 14:20:00 (that includes 10
command processing, format of traces, etc)
as one can see on the chart  the dip in performance is very small, measured
looking at raw data its 8%.
the system runs at ~920k 4k random read iops /sec and the workload running
on the node pushes the system almost to 100% cpu time even without trace
running, reason i tested under this condition is because also on a HPC
compute oriented workload you will run without spare cpu cycles, so the 8%
is under extreme conditions.
said all this, the workload i am running is not causing a lot of metadata
i/o as i am running within a small number of files, its all read, so no
write, therefore this isn't representative to a lot of real world
workloads, but covers one very extreme case , high iops requirements under
heavy cpu contention.
therefore i repeated the test under a heavy metadata workload , SpecSFS
SWBUILD. when i run the test with and without traces turned on i see a ~15%
loss on max operations/sec/node. again this is another extreme case as
there are only few workloads which have a metadata workload component as
heavy as SWBUILD. but given that between the 2 extreme cases we are talking
about 8 - 15% overhead its is very likely that all real world workloads
suffer less than 15% when traces are turned on.
only a test in your environment will really show .

i would not recommend to try this with any version prior 4.2.1 as the
impact will be significant higher than what i measured. hope this helps
make a informed decision.

Sven


On Tue, Mar 7, 2017 at 9:32 PM Oesterlin, Robert <
Robert.Oesterlin at nuance.com> wrote:

> I’m considering enabling trace on all nodes all the time, doing something
> like this:
>
>
>
> mmtracectl --set --trace=def --trace-recycle=global
> --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M
> mmtracectl --start
>
>
>
> My questions are:
>
>
>
> - What is the performance penalty of leaving this on 100% of the time on a
> node?
>
> - Does anyone have any suggestions on automation on stopping trace when a
> particular event occurs?
>
> - What other issues, if any?
>
>
>
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
> 507-269-0413 <(507)%20269-0413>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170308/6e72878b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pasted1
Type: image/png
Size: 216901 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170308/6e72878b/attachment.png>


More information about the gpfsug-discuss mailing list