[Trilinos-Users] Belos GMRES results changing with MPI rank count

[Rutherford, Joseph M]

All,

My custom operator expresses a diagonalized dense operator as a cascade of sparse operators.  If A,B,C are Tpetra::Operators, I is an optional identity Operator effecting Tpetra::Import, and x,y are MultiVectors, then my custom operator is executing y=(A*B+C)*I*x.   My system is

1.)    Vectors x,y are 1:1 distributed in a non-uniform map.

2.)    apply() consistently computes the same mat-vec with MPI rank counts 1 and 2.

3.)    Belos GMRES converges to validated answers with 1 MPI rank.

4.)    Belos GMRES converges to different answers with MPI rank counts 1, 2, and 4.

Points 1 and 2 suggest that perhaps the operator itself is working correctly. I have no idea what I might be doing incorrectly with Belos.  The only "GMRES" solver parameters I'm defining are "Convergence Tolerance" and "Verbosity".  For verbosity=127 (all enums summed together), I get the following output:

<console>
Belos::StatusTestGeneralOutput: Passed
  (Num calls,Mod test,State test): (153, 1, Passed)
   Passed.......OR Combination ->
     OK...........Number of Iterations = 152 < 1000
     Converged....(2-Norm Res Vec) / (2-Norm Prec Res0)
                  residual [ 0 ] = 8.47249e-07 < 1e-06

Passed.......OR Combination ->
  OK...........Number of Iterations = 152 < 1000
  Converged....(2-Norm Res Vec) / (2-Norm Prec Res0)
               residual [ 0 ] = 8.47249e-07 < 1e-06


=========================================================================================================================

                                          TimeMonitor results over 4 processors

Timer Name                                        MinOverProcs    MeanOverProcs    MaxOverProcs    MeanOverCallCounts
-------------------------------------------------------------------------------------------------------------------------
Belos: Operation Op*x                             9.564 (154)     9.717 (154)      9.883 (154)     0.0631 (154)
Belos: Operation Prec*x                           0 (0)           0 (0)            0 (0)           0 (0)
Belos: Orthogonalization                          0.1221 (153)    0.2889 (153)     0.4433 (153)    0.001888 (153)
Belos: PseudoBlockGmresSolMgr total solve time    9.949 (1)       9.949 (1)        9.95 (1)        9.949 (1)
=========================================================================================================================
</console>

Can anyone please suggest how to better diagnose the problem?

Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20160608/1364f6ad/attachment.html>