ADAPTOR System Overview

HPF Performance Analysis with ADAPTOR

ADAPTOR provides several means for performance analysis: timing statistics, communication statistics and tracing.

timing statistics will be generated in the file <executable>.timestat, if the executable is called with option -time,
consisting of a summary of all the routines called at program run with their average time consumptions followed by the timing stastistics of each processor. Example.
For better understanding of the timing statistics it might be useful to have a glance at the callgraph.
communication statistics will be generated in the file <executable>.commstat, if the executable is called with option -comm,
consisting of a summary of Megabytes sent by all processors to all the involved processors followed by detailed communication statistics of each processor. Example.
The options -time and -stat may be combined.

trace statistics with VAMPIRtrace (Visualization and Analysis of MPI Resources):
Linking the MPI program with the VT version of the DALIB will gather information about the execution behavior that can be fed into VAMPIR.

Some example VAMPIR graphics on the hydflo program:

histogram logarithmic scale

Black lines: inter node communication

Example: Callgraph


 fadapt -call hydflo.hpf

will generate a file test.call:



CALL Hierarchy (User Nodes)
===========================
HYDFLO
 . SECOND
 . INITIAL
 .  . GHOST5
 . SEEDRANF
 .  . MPPRANS
 .  .  . RANFK
 .  .  .  . IRANFEVEN
 .  .  . RANFKBINARY
 .  .  .  . IRANFODD
 .  .  . RANFATOK
 .  .  .  . RANFMODMULT
 .  .  . RANFMODMULT (+)
 . MPPRANF
 . GHOST3
 . FLUX
 . GHOST5 (+)
 . HYDRO

Example: Timing statistics

generated by calling the executable with option -time:


 hydflo -time

will generate on stdout after the regular program output:


... timing statistics in file hydflo.timestat.

Contents of file hydflo.timestat (program run on 4 processors):



Timing statistics summary for program hydflo (4 procs) in seconds:

Name of Routine      called   Aver netto +-Max. diff.  Aver brutto +-Max. diff.
                     by #p.
FLUX                    4          1.790 +-     0.028        1.790 +-     0.028
GHOST3                  4          0.150 +-     0.059        0.150 +-     0.059
GHOST5                  4          1.519 +-     0.005        1.519 +-     0.005
HYDFLO                  4          0.910 +-     0.336       10.603 +-     0.058
HYDRO                   4          0.911 +-     0.013        0.911 +-     0.013
INITIAL                 4          0.245 +-     0.174        0.263 +-     0.174
IRANFEVEN               4          0.000 +-     0.000        0.000 +-     0.000
IRANFODD                4          0.004 +-     0.001        0.004 +-     0.001
MPPRANF                 4          1.363 +-     0.063        1.363 +-     0.063
MPPRANS                 4          1.163 +-     0.240        3.191 +-     0.324
RANFATOK                4          0.005 +-     0.003        0.013 +-     0.001
RANFK                   4          0.001 +-     0.001        0.001 +-     0.001
RANFKBINARY             4          0.002 +-     0.001        0.005 +-     0.001
RANFMODMULT             4          2.017 +-     0.086        2.017 +-     0.086
SECOND                  4          0.523 +-     0.460        0.523 +-     0.460
SEEDRANF                4          0.001 +-     0.001        3.192 +-     0.323



Timing statistics of the processes:

Process 1:
Name of Routine       Used time (netto)  Used time (brutto)
FLUX                          1.819 sec           1.819 sec
GHOST3                        0.209 sec           0.209 sec
GHOST5                        1.514 sec           1.514 sec
HYDFLO                        1.246 sec          10.661 sec
HYDRO                         0.924 sec           0.924 sec
INITIAL                       0.071 sec           0.089 sec
IRANFEVEN                     0.000 sec           0.000 sec
IRANFODD                      0.004 sec           0.004 sec
MPPRANF                       1.300 sec           1.300 sec
MPPRANS                       1.403 sec           3.515 sec
RANFATOK                      0.007 sec           0.014 sec
RANFK                         0.000 sec           0.000 sec
RANFKBINARY                   0.001 sec           0.005 sec
RANFMODMULT                   2.100 sec           2.100 sec
SECOND                        0.063 sec           0.063 sec
SEEDRANF                      0.000 sec           3.515 sec

Process 2:
Name of Routine       Used time (netto)  Used time (brutto)
FLUX                          1.782 sec           1.782 sec
GHOST3                        0.160 sec           0.160 sec
GHOST5                        1.520 sec           1.520 sec
HYDFLO                        0.831 sec          10.628 sec
HYDRO                         0.911 sec           0.911 sec
INITIAL                       0.304 sec           0.324 sec
IRANFEVEN                     0.000 sec           0.000 sec
IRANFODD                      0.004 sec           0.004 sec
MPPRANF                       1.367 sec           1.367 sec
MPPRANS                       1.056 sec           3.001 sec
RANFATOK                      0.008 sec           0.013 sec
RANFK                         0.001 sec           0.001 sec
RANFKBINARY                   0.001 sec           0.005 sec
RANFMODMULT                   1.931 sec           1.931 sec
SECOND                        0.752 sec           0.752 sec
SEEDRANF                      0.000 sec           3.001 sec

Process 3:
Name of Routine       Used time (netto)  Used time (brutto)
FLUX                          1.780 sec           1.780 sec
GHOST3                        0.130 sec           0.130 sec
GHOST5                        1.521 sec           1.521 sec
HYDFLO                        0.790 sec          10.562 sec
HYDRO                         0.910 sec           0.910 sec
INITIAL                       0.302 sec           0.319 sec
IRANFEVEN                     0.000 sec           0.000 sec
IRANFODD                      0.003 sec           0.003 sec
MPPRANF                       1.375 sec           1.375 sec
MPPRANS                       0.996 sec           2.993 sec
RANFATOK                      0.003 sec           0.012 sec
RANFK                         0.001 sec           0.001 sec
RANFKBINARY                   0.003 sec           0.006 sec
RANFMODMULT                   1.987 sec           1.987 sec
SECOND                        0.760 sec           0.760 sec
SEEDRANF                      0.001 sec           2.994 sec

Process 4:
Name of Routine       Used time (netto)  Used time (brutto)
FLUX                          1.781 sec           1.781 sec
GHOST3                        0.101 sec           0.101 sec
GHOST5                        1.520 sec           1.520 sec
HYDFLO                        0.772 sec          10.562 sec
HYDRO                         0.900 sec           0.900 sec
INITIAL                       0.302 sec           0.320 sec
IRANFEVEN                     0.000 sec           0.000 sec
IRANFODD                      0.004 sec           0.004 sec
MPPRANF                       1.412 sec           1.412 sec
MPPRANS                       1.195 sec           3.256 sec
RANFATOK                      0.004 sec           0.012 sec
RANFK                         0.000 sec           0.000 sec
RANFKBINARY                   0.002 sec           0.006 sec
RANFMODMULT                   2.051 sec           2.051 sec
SECOND                        0.517 sec           0.517 sec
SEEDRANF                      0.001 sec           3.257 sec

Example: Communication statistics

generated by calling the executable with option -comm:


 hydflo -comm

will generate on stdout after the regular program output:


 Communication Statistics Summary for program hydflo (4 procs):
 Number of Mega-Bytes sent by all procs:
to:       proc1     proc2     proc3     proc4       all
all    3035.367  3036.367  3065.570  3065.617 12202.922
proc1     0.000  1916.688  1149.883     0.000  3066.570
proc2  1915.688     0.000     0.000  1148.930  3064.617
proc3  1119.680     0.000     0.000  1916.688  3036.367
proc4     0.000  1119.680  1915.688     0.000  3035.367
... detailed communication statistics in file hydflo.commstat.

Contents of File hydflo.commstat (program run on 4 processors):



 Number of Mega-Bytes sent by all procs:
to:       proc1     proc2     proc3     proc4       all
all    3035.367  3036.367  3065.570  3065.617 12202.922
proc1     0.000  1916.688  1149.883     0.000  3066.570
proc2  1915.688     0.000     0.000  1148.930  3064.617
proc3  1119.680     0.000     0.000  1916.688  3036.367
proc4     0.000  1119.680  1915.688     0.000  3035.367
 Number of sends of all procs:
to:       proc1     proc2     proc3     proc4       all
all        1498      1748      1181      1187      5614
proc1         0       839       592         0      1431
proc2       589         0         0       348       937
proc3       909         0         0       839      1748
proc4         0       909       589         0      1498
 Number of Mega-Bytes received by all procs:
from:     proc1     proc2     proc3     proc4       all
all    3066.570  3064.617  3036.367  3035.367 12202.922
proc1     0.000  1915.688  1119.680     0.000  3035.367
proc2  1916.688     0.000     0.000  1119.680  3036.367
proc3  1149.883     0.000     0.000  1915.688  3065.570
proc4     0.000  1148.930  1916.688     0.000  3065.617
 Number of receives of all procs:
from:     proc1     proc2     proc3     proc4       all
all        1431       937      1748      1498      5614
proc1         0       589       909         0      1498
proc2       839         0         0       909      1748
proc3       592         0         0       589      1181
proc4         0       348       839         0      1187
Communication statistics summary of all procs:
comm. with:        proc1     proc2     proc3     proc4       all
# sends to:         1498      1748      1181      1187      5614
total Mbytes:   3035.367  3036.367  3065.570  3065.617 12202.922
minl.  sent:       0.008     0.004     0.004     0.004     0.004
maxl.  sent:      11.250    11.250    11.250    11.250    11.250
averl. sent:       2.026     1.737     2.596     2.583     2.174
# recs from:        1431       937      1748      1498      5614
total Mbytes:   3066.570  3064.617  3036.367  3035.367 12202.922
minl.  rec'd:      0.004     0.008     0.004     0.008     0.004
maxl.  rec'd:     11.250    11.250    11.250    11.250    11.250
averl. rec'd:      2.143     3.271     1.737     2.026     2.174
Communication statistics of proc # 1:
comm. with:       proc1    proc2    proc3    proc4      all
# sends to:           0      839      592        0     1431
total Mbytes:     0.000 1916.688 1149.883    0.000 3066.570
minl.  sent:      0.000    0.004    0.004    0.000    0.004
maxl.  sent:      0.000   11.250   11.250    0.000   11.250
averl. sent:      0.000    2.284    1.942    0.000    2.143
# recs from:          0      589      909        0     1498
total Mbytes:     0.000 1915.688 1119.680    0.000 3035.367
minl.  rec'd:     0.000    0.008    0.008    0.000    0.008
maxl.  rec'd:     0.000   11.250   11.250    0.000   11.250
averl. rec'd:     0.000    3.252    1.232    0.000    2.026
Communication statistics of proc # 2:
comm. with:       proc1    proc2    proc3    proc4      all
# sends to:         589        0        0      348      937
total Mbytes:  1915.688    0.000    0.000 1148.930 3064.617
minl.  sent:      0.008    0.000    0.000    0.008    0.008
minl.  sent:      0.008    0.000    0.000    0.008    0.008
maxl.  sent:     11.250    0.000    0.000   11.250   11.250
averl. sent:      3.252    0.000    0.000    3.302    3.271
# recs from:        839        0        0      909     1748
total Mbytes:  1916.688    0.000    0.000 1119.680 3036.367
minl.  rec'd:     0.004    0.000    0.000    0.008    0.004
maxl.  rec'd:    11.250    0.000    0.000   11.250   11.250
averl. rec'd:     2.284    0.000    0.000    1.232    1.737
Communication statistics of proc # 3:
comm. with:       proc1    proc2    proc3    proc4      all
# sends to:         909        0        0      839     1748
total Mbytes:  1119.680    0.000    0.000 1916.688 3036.367
minl.  sent:      0.008    0.000    0.000    0.004    0.004
maxl.  sent:     11.250    0.000    0.000   11.250   11.250
averl. sent:      1.232    0.000    0.000    2.284    1.737
# recs from:        592        0        0      589     1181
total Mbytes:  1149.883    0.000    0.000 1915.688 3065.570
minl.  rec'd:     0.004    0.000    0.000    0.008    0.004
maxl.  rec'd:    11.250    0.000    0.000   11.250   11.250
averl. rec'd:     1.942    0.000    0.000    3.252    2.596
Communication statistics of proc # 4:
comm. with:       proc1    proc2    proc3    proc4      all
# sends to:           0      909      589        0     1498
total Mbytes:     0.000 1119.680 1915.688    0.000 3035.367
minl.  sent:      0.000    0.008    0.008    0.000    0.008
maxl.  sent:      0.000   11.250   11.250    0.000   11.250
averl. sent:      0.000    1.232    3.252    0.000    2.026
# recs from:          0      348      839        0     1187
total Mbytes:     0.000 1148.930 1916.688    0.000 3065.617
minl.  rec'd:     0.000    0.008    0.004    0.000    0.004
maxl.  rec'd:     0.000   11.250   11.250    0.000   11.250
averl. rec'd:     0.000    3.302    2.284    0.000    2.583