MueLu - Level smoothers

General remarks

  • MueLu uses algorithms from other Trilinos packages for level smoothers. These include Ifpack/Ifpack2 and Teko. Similar for direct solvers: MueLu uses the direct solver provided by Amesos/Amesos2.
  • For creating concrete smoothers on the different levels we use the prototype pattern. I.e. we define one (or more) smoother prototypes which are given to the SmootherFactory class responsible for the concrete level.

Level smoothers in MueLu

General preparations

Declare location of include headers


In [1]:
.I /opt/install/do-conf-ep-serial/include/


Out[1]:

Load Trilinos libraries


In [2]:
.L libepetra


Out[2]:


In [3]:
.L libxpetra


Out[3]:


In [4]:
.L libmuelu


Out[4]:


In [5]:
.L libteuchoscore


Out[5]:

Include some standard header files to create a communicator and an Xpetra::Map


In [6]:
#include "Kokkos_DefaultNode.hpp"
#include "Epetra_SerialComm.h"
#include "Epetra_Vector.h"
#include "Epetra_CrsMatrix.h"
#include "Teuchos_RCP.hpp"
#include "Teuchos_DefaultSerialComm.hpp"
#include "Xpetra_MapFactory.hpp"
#include "Xpetra_Map.hpp"
#include "Xpetra_EpetraCrsMatrix.hpp"


Out[6]:

Again, we use standard template parameters:


In [7]:
typedef double Scalar;
typedef int LocalOrdinal;
typedef int GlobalOrdinal;
typedef Kokkos::Compat::KokkosSerialWrapperNode EpetraNode;
typedef EpetraNode Node;
typedef Scalar SC;
typedef LocalOrdinal LO;
typedef GlobalOrdinal GO;
typedef Node NO;


Out[7]:


In [8]:
#include "Xpetra_CrsMatrixWrap.hpp"
#include "Xpetra_CrsMatrix.hpp"
#include "Xpetra_MultiVector.hpp"
#include "Xpetra_BlockedMultiVector.hpp"
#include "Xpetra_BlockedVector.hpp"
#include "Xpetra_MultiVectorFactory.hpp"


Out[8]:

Create a matrix object

Let's create a matrix and wrap it as an Xpetra::CrsMatrixWrap object


In [9]:
int NumMyElements = 30;


Out[9]:
(int) 30

In [10]:
Epetra_SerialComm Comm;
Epetra_Map Map(-1, NumMyElements, 0, Comm);
int NumGlobalElements = Map.NumGlobalElements();

Epetra_CrsMatrix A(Copy, Map, 3);

double negOne = -1.0;
double posTwo = 2.0;
double posOne = +1.0;
double zero   = 0.0;
for (int i=0; i<NumMyElements; i++) {
    int GlobalRow = A.GRID(i); int RowLess1 = GlobalRow - 1; int RowPlus1 = GlobalRow + 1;

    if (RowLess1!=-1) A.InsertGlobalValues(GlobalRow, 1, &negOne, &RowLess1);
    if (RowPlus1!=NumGlobalElements) A.InsertGlobalValues(GlobalRow, 1, &negOne, &RowPlus1);
    A.InsertGlobalValues(GlobalRow, 1, &posTwo, &GlobalRow);
};

int idxZero = 0;
int idxOne  = 1;
A.ReplaceGlobalValues(0,1,&posOne, &idxZero);
A.ReplaceGlobalValues(0,1,&zero,&idxOne);


A.FillComplete();


Out[10]:
(int) 0

In [11]:
Teuchos::RCP<Epetra_CrsMatrix> rcpA = Teuchos::rcpFromRef(A);
Xpetra::EpetraCrsMatrixT<GO,NO> xA(rcpA);
Teuchos::RCP<Xpetra::CrsMatrix<SC,LO,GO,NO> > xCrsMat = Teuchos::rcpFromRef(xA);
Xpetra::CrsMatrixWrap<SC,LO,GO,NO> xCrsWrap = Xpetra::CrsMatrixWrap<SC,LO,GO,NO>(xCrsMat);
Teuchos::RCP<Xpetra::Matrix<SC,LO,GO,NO> > xMat = Teuchos::rcpFromRef(xCrsWrap);


Out[11]:
(Teuchos::RCP<Xpetra::Matrix<SC, LO, GO, NO> > &) @0x7f4a182be318

The xCrsWrap object contains the matrix object that can be used as input for MueLu.

Create a smoother

Smoother details are stored in a Teuchos::ParameterList. For demonstration purposes we create a Symmetric Gauss-Seidel smoother ($50$ sweeps, damping factor $0.45$):


In [12]:
#include "Teuchos_ParameterList.hpp"


Out[12]:


In [13]:
Teuchos::ParameterList smooParams;


Out[13]:
(Teuchos::ParameterList &) {}

In [14]:
smooParams.set("type", Teuchos::ParameterEntry("RELAXATION"));
Teuchos::ParameterList& subSmooParams = smooParams.sublist("ParamterList");
subSmooParams.set("relaxation: type", "Symmetric Gauss-Seidel");
subSmooParams.set("relaxation: sweeps",50);
subSmooParams.set("relaxation: damping factor",0.45);


Out[14]:
(Teuchos::ParameterList &) { "relaxation: type" => @0xc1fda48, "relaxation: sweeps" => @0xc1fda90, "relaxation: damping factor" => @0xc1fdad8 }

You can easily print the content of a parameter list using


In [15]:
std::cout << smooParams << std::endl;


type = RELAXATION   [unused]   [unused]
ParamterList -> 
 relaxation: type = Symmetric Gauss-Seidel   [unused]
 relaxation: sweeps = 50   [unused]
 relaxation: damping factor = 0.45   [unused]

Out[15]:
(std::basic_ostream<char, std::char_traits<char> >::__ostream_type &) @0x7f4a10003700

The TrilinosSmoother class is a wrapper class for most relaxation-based and ILU type smoothers (provided by Ifpack or Ifpack2):


In [16]:
#include "MueLu_TrilinosSmoother.hpp"


Out[16]:

To create a new TrilinosSmoother you have to provide a type (e.g., RELAXATION) and the corresponding Ifpack/Ifpack2 parameter list:


In [17]:
MueLu::TrilinosSmoother<SC,LO,GO,NO> trSmoother("RELAXATION", subSmooParams);


Out[17]:
(MueLu::TrilinosSmoother<SC, LO, GO, NO> &) @0x7f4a182be3e0

The smoother use variable A stored on the Level class as input. Therefore, let's create a Level container and store the matrix:


In [18]:
#include "MueLu_Level.hpp"


Out[18]:


In [19]:
MueLu::Level l;


Out[19]:
(MueLu::Level &) @0x7f4a182be5c0
Note:It is important, that ```A``` is a ```RCP>``` object for the smoother to be able to properly cast the matrix object (and determine whether we are using the *Epetra* or *Tpetra* stack).

In [20]:
l.Set("A",xMat);


Out[20]:
(void) @0x7ffda227d1d0

Furthermore, we have to store the type of the underlying linear algebra package used underneath in the Level class. This is necessary such that the TrilinosSmoother class can distinguish between Epetra and Tpetra and create either Ifpack or Ifpack2 preconditioners.

Note:We have to explicitly set this flag here. When using the standard MueLu setup routines, this is automatically done and the user does not have to think about that.

In [21]:
l.setlib(Xpetra::UseEpetra);


Out[21]:
(void) @0x7ffda227d1d0

Then, we can request the Smoother. Without requesting the smoother the smoother would not be built.


In [22]:
l.Request("Smoother", &trSmoother);


Out[22]:
(void) @0x7ffda227d1d0

In [23]:
l.print(std::cout, MueLu::Extreme);


LevelID = -1
  data name             gen. factory addr.  req    keep   type             data            req'd by            
  --------------------  ------------------  -----  -----  ---------------  --------------  --------------------
  A                     NoFactory           1      User   Matrix           available       0x11f22808          
  Smoother              0x7f4a182be3e8      1      No     unknown          not available   0x123b69e0          
Out[23]:
(void) @0x7ffda227d1d0

Then we can call the Setup routine of the smoother object. Depending on the smoother type, the ILU triangle block are built or the diagonal is inverted. We only have to provide the current level container with A stored.


In [24]:
trSmoother.Setup(l);


Setup Smoother (MueLu::IfpackSmoother{type = point relaxation stand-alone})
 IFPACK (Local SGS, sweeps=50, damping=0.45)
Out[24]:
(void) @0x7ffda227d1d0

For using the smoother we need a solution guess and a RHS vector.


In [25]:
Teuchos::RCP<Xpetra::MultiVector<SC,LO,GO,NO>> x = Xpetra::MultiVectorFactory<SC,LO,GO,NO>::Build(xMat->getRowMap(),1);


Out[25]:
(Teuchos::RCP<Xpetra::MultiVector<SC, LO, GO, NO> > &) @0x7f4a182be6c0

In [26]:
Teuchos::RCP<Xpetra::MultiVector<SC,LO,GO,NO>> b = Xpetra::MultiVectorFactory<SC,LO,GO,NO>::Build(xMat->getRowMap(),1);


Out[26]:
(Teuchos::RCP<Xpetra::MultiVector<SC, LO, GO, NO> > &) @0x7f4a182be6d8

In [27]:
b->putScalar(1.0);


Out[27]:
(void) @0x7ffda227d1d0

Then we can call the Apply routine. In our example it performs $50$ SGS sweeps (damping $0.45$):


In [28]:
trSmoother.Apply(*x,*b,true);


Out[28]:
(void) @0x7ffda227d1d0

Then you print the solution vector and the matrix...


In [29]:
std::cout << *x << std::endl;


     MyPID           GID               Value  
         0             0                       1
         0             1                 6.49166
         0             2                 11.0806
         0             3                 14.8804
         0             4                 17.9917
         0             5                   20.51
         0             6                 22.5236
         0             7                 24.1134
         0             8                 25.3514
         0             9                 26.3004
         0            10                 27.0142
         0            11                 27.5373
         0            12                 27.9054
         0            13                 28.1456
         0            14                 28.2765
         0            15                 28.3089
         0            16                 28.2461
         0            17                 28.0834
         0            18                 27.8084
         0            19                 27.4012
         0            20                 26.8334
         0            21                 26.0683
         0            22                 25.0607
         0            23                 23.7564
         0            24                 22.0922
         0            25                 19.9963
         0            26                 17.3887
         0            27                 14.1817
         0            28                 10.2813
         0            29                 5.58836

Out[29]:
(std::basic_ostream<char, std::char_traits<char> >::__ostream_type &) @0x7f4a10003700

In [30]:
std::cout << A << std::endl;


Epetra::CrsMatrix
Number of Global Rows        = 30
Number of Global Cols        = 30
Number of Global Diagonals   = 30
Number of Global Nonzeros    = 88
Global Maximum Num Entries   = 3





Number of My Rows        = 30
Number of My Cols        = 30
Number of My Diagonals   = 30
Number of My Nonzeros    = 88
My Maximum Num Entries   = 3

   Processor    Row Index    Col Index           Value     
       0             0             0                       1    
       0             0             1                       0    
       0             1             0                      -1    
       0             1             1                       2    
       0             1             2                      -1    
       0             2             1                      -1    
       0             2             2                       2    
       0             2             3                      -1    
       0             3             2                      -1    
       0             3             3                       2    
       0             3             4                      -1    
       0             4             3                      -1    
       0             4             4                       2    
       0             4             5                      -1    
       0             5             4                      -1    
       0             5             5                       2    
       0             5             6                      -1    
       0             6             5                      -1    
       0             6             6                       2    
       0             6             7                      -1    
       0             7             6                      -1    
       0             7             7                       2    
       0             7             8                      -1    
       0             8             7                      -1    
       0             8             8                       2    
       0             8             9                      -1    
       0             9             8                      -1    
       0             9             9                       2    
       0             9            10                      -1    
       0            10             9                      -1    
       0            10            10                       2    
       0            10            11                      -1    
       0            11            10                      -1    
       0            11            11                       2    
       0            11            12                      -1    
       0            12            11                      -1    
       0            12            12                       2    
       0            12            13                      -1    
       0            13            12                      -1    
       0            13            13                       2    
       0            13            14                      -1    
       0            14            13                      -1    
       0            14            14                       2    
       0            14            15                      -1    
       0            15            14                      -1    
       0            15            15                       2    
       0            15            16                      -1    
       0            16            15                      -1    
       0            16            16                       2    
       0            16            17                      -1    
       0            17            16                      -1    
       0            17            17                       2    
       0            17            18                      -1    
       0            18            17                      -1    
       0            18            18                       2    
       0            18            19                      -1    
       0            19            18                      -1    
       0            19            19                       2    
       0            19            20                      -1    
       0            20            19                      -1    
       0            20            20                       2    
       0            20            21                      -1    
       0            21            20                      -1    
       0            21            21                       2    
       0            21            22                      -1    
       0            22            21                      -1    
       0            22            22                       2    
       0            22            23                      -1    
       0            23            22                      -1    
       0            23            23                       2    
       0            23            24                      -1    
       0            24            23                      -1    
       0            24            24                       2    
       0            24            25                      -1    
       0            25            24                      -1    
       0            25            25                       2    
       0            25            26                      -1    
       0            26            25                      -1    
       0            26            26                       2    
       0            26            27                      -1    
       0            27            26                      -1    
       0            27            27                       2    
       0            27            28                      -1    
       0            28            27                      -1    
       0            28            28                       2    
       0            28            29                      -1    
       0            29            28                      -1    
       0            29            29                       2    

Out[30]:
(std::basic_ostream<char, std::char_traits<char> >::__ostream_type &) @0x7f4a10003700

... or you calculate the Residual or Residual norm:


In [31]:
#include "Teuchos_Array.hpp"


Out[31]:


In [32]:
#include "MueLu_Utilities.hpp"


Out[32]:


In [33]:
MueLu::Utilities<SC,LO,GO,NO>::ResidualNorm(*xMat,*x,*b);


Out[33]:
(Teuchos::Array<MueLu::Utilities<double, int, int, Kokkos::Compat::KokkosSerialWrapperNode>::Magnitude>) { 3.606524 }

Create a direct solver object

In a similar way, we can define a direct solver. As type, we choose a direct solver from Amesos/Amesos2 such as KLU and a parameter list. In our example an empty parameter list is sufficient.


In [34]:
#include "MueLu_DirectSolver.hpp"


Out[34]:


In [35]:
Teuchos::ParameterList dsPl;
MueLu::DirectSolver<SC,LO,GO,NO> ds("Klu", dsPl);


Out[35]:
(MueLu::DirectSolver<SC, LO, GO, NO> &) @0x7f4a182be790

In [36]:
x->putScalar(0.0);


Out[36]:
(void) @0x7ffda227d1d0

Before we call the Setup routine we have to request the DirectSolver on the level class.


In [37]:
l.Request("DirectSolver", &ds);


Out[37]:
(void) @0x7ffda227d1d0

Then, it is time for the Setup call which usually performs the factorization.


In [38]:
ds.Setup(l);


Setup Smoother (MueLu::AmesosSmoother{type = Klu})
Out[38]:
(void) @0x7ffda227d1d0

Now, we can call the Apply routine and solve the problem (exactly).


In [39]:
ds.Apply(*x,*b,true);


Out[39]:
(void) @0x7ffda227d1d0

Again, we can print out the solution and calculate the residual norm:


In [40]:
std::cout << *x << std::endl;


     MyPID           GID               Value  
         0             0                       1
         0             1                 15.4667
         0             2                 28.9333
         0             3                    41.4
         0             4                 52.8667
         0             5                 63.3333
         0             6                    72.8
         0             7                 81.2667
         0             8                 88.7333
         0             9                    95.2
         0            10                 100.667
         0            11                 105.133
         0            12                   108.6
         0            13                 111.067
         0            14                 112.533
         0            15                     113
         0            16                 112.467
         0            17                 110.933
         0            18                   108.4
         0            19                 104.867
         0            20                 100.333
         0            21                    94.8
         0            22                 88.2667
         0            23                 80.7333
         0            24                    72.2
         0            25                 62.6667
         0            26                 52.1333
         0            27                    40.6
         0            28                 28.0667
         0            29                 14.5333

Out[40]:
(std::basic_ostream<char, std::char_traits<char> >::__ostream_type &) @0x7f4a10003700

In [41]:
MueLu::Utilities<SC,LO,GO,NO>::ResidualNorm(*xMat,*x,*b);


Out[41]:
(Teuchos::Array<MueLu::Utilities<double, int, int, Kokkos::Compat::KokkosSerialWrapperNode>::Magnitude>) { 0.000000 }

Level smoothers in a MueLu hierarchy

In order to make the smoothers available to the multigrid hierarchy, we have to create

  1. the smoothers
  2. Create a SmootherFactory class accepting the smoother objects for pre- and/or postsmoothing
  3. Register the SmootherFactory object in a FactoryManager

More details below and in the Hierarchy lesson.

First, we include the necessary headers for the MueLu::Hierarchy


In [48]:
#include "MueLu_MutuallyExclusiveTime.hpp"
#include "MueLu_TimeMonitor.hpp"


Out[48]:


In [50]:
#include "MueLu_Hierarchy.hpp"


Out[50]:

Then, we create a new MueLu::Hierarchy with standard multigrid parameters.


In [51]:
MueLu::Hierarchy<SC,LO,GO,NO> H(xMat);


Out[51]:
(MueLu::Hierarchy<SC, LO, GO, NO> &) @0x7f4a182be950

We set the maximum size for the coarsest level to $10$ and increase the verbosity output level to get more screen output.


In [52]:
H.setDefaultVerbLevel(Teuchos::VERB_HIGH);
H.SetMaxCoarseSize(10);


Out[52]:
(void) @0x7ffda227d1d0
SmootherFactory

Then, we create a MueLu::SmootherFactory which serves as general interface for the smoothers and direct solvers for the multigrid hierarchy.


In [53]:
#include "MueLu_SmootherFactory.hpp"


Out[53]:


In [54]:
MueLu::SmootherFactory<SC,LO,GO,NO> smootherFact;


Out[54]:
(MueLu::SmootherFactory<SC, LO, GO, NO> &) @0x7f4a182bead8

We fill the SmootherFactory with a concrete smoother. In this case, we use the trSmoo smoother object for pre- and post smoothing. One could also provide two distinct smoothers/solvers for pre-/postsmoothing or use the MueLu::NoSmoother


In [55]:
MueLu::TrilinosSmoother<SC,LO,GO,NO> trSmoo("RELAXATION", subSmooParams);


Out[55]:
(MueLu::TrilinosSmoother<SC, LO, GO, NO> &) @0x7f4a182bec50

In [56]:
smootherFact.SetSmootherPrototypes(Teuchos::rcpFromRef(trSmoo));


Out[56]:
(void) @0x7ffda227d1d0

The FactoryManager

Then, we create a FactoryManager which manages the default instances for each variable. More details on the FactoryManager can be found in the Hierarchy lesson.


In [57]:
#include "MueLu_FactoryManager.hpp"


Out[57]:


In [58]:
MueLu::FactoryManager<SC,LO,GO,NO> M;


Out[58]:
(MueLu::FactoryManager<SC, LO, GO, NO> &) @0x7f4a182bee30

Use the SetFactory routine to register the SmootherFactory to produce the Smoother variable (and the DirectSolver).

Note: For real examples one would use a different factory providing an actual direct solver from the *Amesos* package, of course.

In [59]:
M.SetFactory("Smoother", Teuchos::rcpFromRef(smootherFact));
M.SetFactory("CoarseSolver", Teuchos::rcpFromRef(smootherFact));


Out[59]:
(void) @0x7ffda227d1d0

With these factories set in the FactoryManager you can create a hierarchy (assuming that all other necessary variables and Factories are properly defined; please see the corresponding notebook on MueLu::Hierarchy).