Strange behavior of a boundary condition loss on the test sample

[kyouma]

Hello. I apologize if there have been a similar topic already, but I have not found it in these issues.

I have been experimenting with a simple time-dependent equation in a rectangular 2D domain, and have found out a strange behavior of BC losses. Or maybe I am doing something wrong.

The image of the PDE system

Below is the code divided into blocks:

Imports and random seed setting

import os
os.environ['DDE_BACKEND'] = 'pytorch'
os.environ['CUDA_VISIBLE_DEVICES'] = '0'


import random
import types

import deepxde as dde
import matplotlib.pyplot as plt
import numpy as np
import torch
import tqdm
import tqdm.auto


random.seed(0)
np.random.seed(0)
dde.config.set_random_seed(0)

Domain limits

LEFT = 0
RIGHT = np.pi
BOTTOM = -np.pi
TOP = np.pi
T = 1.0

PDE, BC, IC, derivatives and exact solution functions

def pde(xyt, u):
    x = xyt[:, 0:1]
    y = xyt[:, 1:2]
    t = xyt[:, 2:3]

    du_t = dde.grad.jacobian(u, xyt, i=0, j=2)
    du_xx = dde.grad.hessian(u, xyt, i=0, j=0)
    du_yy = dde.grad.hessian(u, xyt, i=1, j=1)

    f = torch.exp(-1.3* t) * (77/250 * torch.sin(0.6 * x) + 3807/400 * torch.cos(2.5 * y))

    return du_t - 1.9 * (du_xx + du_yy) - f


def u_torch(xyt):
    '''Same as u_numpy(), but written with PyTorch.'''
    if not isinstance(xyt, torch.Tensor):
        xyt = torch.tensor(xyt)
    x = xyt[:, 0:1]
    y = xyt[:, 1:2]
    t = xyt[:, 2:3]
    return torch.exp(-1.3 * t) * (0.5 * torch.sin(0.6 * x) + 0.9 * torch.cos(2.5 * y)) + 2.4


def u_numpy(xyt):
    '''Same as u_torch(), but written with NumPy.'''
    if isinstance(xyt, torch.Tensor):
        xyt = xyt.detach().cpu().numpy()
    x = xyt[:, 0:1]
    y = xyt[:, 1:2]
    t = xyt[:, 2:3]
    return np.exp(-1.3 * t) * (0.5 * np.sin(0.6 * x) + 0.9 * np.cos(2.5 * y)) + 2.4


def u_ic(xyt):
    if isinstance(xyt, torch.Tensor):
        xyt = xyt.detach().cpu().numpy()
    x = xyt[:, 0:1]
    y = xyt[:, 1:2]
    t = xyt[:, 2:3]
    return 0.5 * np.sin(0.6 * x) + 0.9 * np.cos(2.5 * y) + 2.4


def du_dx(xyt):
    if isinstance(xyt, torch.Tensor):
        xyt = xyt.detach().cpu().numpy()
    x = xyt[:, 0:1]
    y = xyt[:, 1:2]
    t = xyt[:, 2:3]
    return 0.3 * np.exp(-1.3 * t) * np.cos(0.6 * x)


def du_dy(xyt):
    if isinstance(xyt, torch.Tensor):
        xyt = xyt.detach().cpu().numpy()
    x = xyt[:, 0:1]
    y = xyt[:, 1:2]
    t = xyt[:, 2:3]
    return -2.25 * np.exp(-1.3 * t) * np.sin(2.5 * y)

on_boundary checking functions with corner points filters

def on_boundary_left(xyt, on_boundary):
    x = xyt[0]
    y = xyt[1]
    t = xyt[2]
    return on_boundary and dde.utils.isclose(x, LEFT) and not (dde.utils.isclose(y, TOP) or dde.utils.isclose(y, BOTTOM))


def on_boundary_right(xyt, on_boundary):
    x = xyt[0]
    y = xyt[1]
    t = xyt[2]
    return on_boundary and dde.utils.isclose(x, RIGHT) and not (dde.utils.isclose(y, TOP) or dde.utils.isclose(y, BOTTOM))


def on_boundary_top(xyt, on_boundary):
    x = xyt[0]
    y = xyt[1]
    t = xyt[2]
    return on_boundary and dde.utils.isclose(y, TOP) and not (dde.utils.isclose(x, LEFT) or dde.utils.isclose(x, RIGHT))


def on_boundary_bottom(xyt, on_boundary):
    x = xyt[0]
    y = xyt[1]
    t = xyt[2]
    return on_boundary and dde.utils.isclose(y, BOTTOM) and not (dde.utils.isclose(x, LEFT) or dde.utils.isclose(x, RIGHT))

Geometry, boundary and initial conditions

geom = dde.geometry.Rectangle([LEFT, BOTTOM], [RIGHT, TOP])
timedomain = dde.geometry.TimeDomain(0, T)
geomtime = dde.geometry.GeometryXTime(geom, timedomain)

nbc_left = dde.icbc.NeumannBC(geomtime, lambda xyt: du_dx(xyt) * (-1), on_boundary_left)  # Correction for the normal direction
nbc_right = dde.icbc.NeumannBC(geomtime, du_dx, on_boundary_right)
dbc_bottom = dde.icbc.DirichletBC(geomtime, u_numpy, on_boundary_bottom)
nbc_top = dde.icbc.NeumannBC(geomtime, du_dy, on_boundary_top)

ic = dde.icbc.IC(geomtime, u_ic, lambda _, on_initial: on_initial)

TimePDE object, neural network and model wrapper

data = dde.data.TimePDE(
    geomtime,
    pde,
    [
        nbc_left,
        nbc_right,
        dbc_bottom,
        nbc_top,        
        ic,
    ],
    num_domain=5000,
    num_boundary=5000,
    num_initial=5000,
    solution=u_numpy,
    num_test=1000,
)

layer_size = [3] + [32] * 4 + [1]
activation = 'tanh'
initializer = 'Glorot uniform'
net = dde.nn.FNN(layer_size, activation, initializer)

model = dde.Model(data, net)

Compile and run

model.compile(
    'adam',
    lr=1e-3,
    loss='MSE',
    metrics=['mean squared error', 'l2 relative error'],
    loss_weights=[1, 1, 1, 1, 1, 1],
)

losshistory, train_state = model.train(
    iterations=10_200,
    display_every=200,
    callbacks=[
        dde.callbacks.PDEPointResampler(period=1000, bc_points=True and dde.backend.backend_name == 'pytorch'),
    ]
)

When PDEPointResampler is disabled, everything's OK: the test losses and train losses have the same scale, the metrics seem to be alright (for LR=1e-3).

Training log when PDEPointResampler is disabled

Training model...

Step      Train loss                                                      Test loss                                                       Test metric             
0         [2.47e+01, 6.13e-01, 4.66e-02, 2.29e+00, 1.27e+00, 4.02e+00]    [1.51e+01, 6.13e-01, 4.66e-02, 2.29e+00, 1.27e+00, 4.02e+00]    [4.01e+00, 7.23e-01]    
200       [1.41e+00, 4.07e-02, 3.24e-02, 1.20e-01, 1.07e-02, 4.25e-01]    [7.80e-01, 4.07e-02, 3.24e-02, 1.20e-01, 1.07e-02, 4.25e-01]    [2.71e-01, 1.88e-01]    
400       [4.45e-02, 3.35e-03, 3.68e-03, 4.82e-03, 5.70e-03, 6.48e-03]    [4.22e-02, 3.35e-03, 3.68e-03, 4.82e-03, 5.70e-03, 6.48e-03]    [1.52e-02, 4.45e-02]    
600       [1.63e-02, 2.23e-03, 1.35e-03, 2.39e-03, 1.92e-03, 3.83e-03]    [1.67e-02, 2.23e-03, 1.35e-03, 2.39e-03, 1.92e-03, 3.83e-03]    [1.23e-02, 4.00e-02]    
800       [1.00e-02, 1.60e-03, 6.82e-04, 1.55e-03, 8.09e-04, 2.26e-03]    [9.13e-03, 1.60e-03, 6.82e-04, 1.55e-03, 8.09e-04, 2.26e-03]    [1.11e-02, 3.81e-02]    
1000      [7.20e-03, 1.17e-03, 4.14e-04, 1.12e-03, 4.42e-04, 1.34e-03]    [5.96e-03, 1.17e-03, 4.14e-04, 1.12e-03, 4.42e-04, 1.34e-03]    [1.06e-02, 3.72e-02]    
1200      [5.76e-03, 8.92e-04, 2.74e-04, 8.05e-04, 3.09e-04, 8.54e-04]    [4.42e-03, 8.92e-04, 2.74e-04, 8.05e-04, 3.09e-04, 8.54e-04]    [1.03e-02, 3.67e-02]    
1400      [1.00e-02, 6.79e-04, 2.20e-04, 6.86e-04, 3.24e-04, 8.32e-04]    [5.89e-03, 6.79e-04, 2.20e-04, 6.86e-04, 3.24e-04, 8.32e-04]    [1.10e-02, 3.79e-02]    
1600      [3.90e-03, 5.66e-04, 1.59e-04, 5.20e-04, 2.35e-04, 4.24e-04]    [3.35e-03, 5.66e-04, 1.59e-04, 5.20e-04, 2.35e-04, 4.24e-04]    [1.03e-02, 3.66e-02]    
1800      [2.92e-03, 4.59e-04, 1.31e-04, 3.44e-04, 2.13e-04, 3.30e-04]    [2.67e-03, 4.59e-04, 1.31e-04, 3.44e-04, 2.13e-04, 3.30e-04]    [1.02e-02, 3.65e-02]    
2000      [2.43e-03, 3.85e-04, 1.13e-04, 2.61e-04, 1.99e-04, 2.60e-04]    [2.38e-03, 3.85e-04, 1.13e-04, 2.61e-04, 1.99e-04, 2.60e-04]    [1.02e-02, 3.64e-02]    
2200      [2.10e-03, 3.26e-04, 1.05e-04, 1.96e-04, 1.85e-04, 2.16e-04]    [2.04e-03, 3.26e-04, 1.05e-04, 1.96e-04, 1.85e-04, 2.16e-04]    [1.02e-02, 3.65e-02]    
2400      [1.47e-02, 2.77e-04, 9.92e-05, 8.40e-04, 1.85e-04, 5.94e-04]    [5.17e-03, 2.77e-04, 9.92e-05, 8.40e-04, 1.85e-04, 5.94e-04]    [1.02e-02, 3.64e-02]    
2600      [1.56e-03, 2.54e-04, 9.24e-05, 1.35e-04, 1.66e-04, 1.51e-04]    [1.60e-03, 2.54e-04, 9.24e-05, 1.35e-04, 1.66e-04, 1.51e-04]    [1.02e-02, 3.64e-02]    
2800      [1.49e-03, 2.28e-04, 9.11e-05, 1.18e-04, 1.58e-04, 1.33e-04]    [1.44e-03, 2.28e-04, 9.11e-05, 1.18e-04, 1.58e-04, 1.33e-04]    [1.03e-02, 3.67e-02]    
3000      [1.24e-03, 2.09e-04, 8.64e-05, 1.02e-04, 1.50e-04, 1.16e-04]    [1.28e-03, 2.09e-04, 8.64e-05, 1.02e-04, 1.50e-04, 1.16e-04]    [1.02e-02, 3.64e-02]    
3200      [1.25e-03, 1.91e-04, 8.42e-05, 9.19e-05, 1.42e-04, 1.11e-04]    [1.11e-03, 1.91e-04, 8.42e-05, 9.19e-05, 1.42e-04, 1.11e-04]    [1.02e-02, 3.64e-02]    
3400      [1.38e-03, 1.80e-04, 8.03e-05, 9.97e-05, 1.39e-04, 1.14e-04]    [1.27e-03, 1.80e-04, 8.03e-05, 9.97e-05, 1.39e-04, 1.14e-04]    [1.01e-02, 3.62e-02]    
3600      [9.33e-04, 1.66e-04, 7.85e-05, 7.66e-05, 1.30e-04, 8.64e-05]    [9.51e-04, 1.66e-04, 7.85e-05, 7.66e-05, 1.30e-04, 8.64e-05]    [1.03e-02, 3.66e-02]    
3800      [8.96e-04, 1.56e-04, 7.63e-05, 7.26e-05, 1.22e-04, 8.32e-05]    [9.11e-04, 1.56e-04, 7.63e-05, 7.26e-05, 1.22e-04, 8.32e-05]    [1.02e-02, 3.65e-02]    
4000      [8.08e-04, 1.47e-04, 7.39e-05, 6.76e-05, 1.17e-04, 7.53e-05]    [8.14e-04, 1.47e-04, 7.39e-05, 6.76e-05, 1.17e-04, 7.53e-05]    [1.03e-02, 3.67e-02]    
4200      [7.85e-04, 1.39e-04, 7.15e-05, 6.59e-05, 1.11e-04, 7.08e-05]    [7.96e-04, 1.39e-04, 7.15e-05, 6.59e-05, 1.11e-04, 7.08e-05]    [1.04e-02, 3.67e-02]    
4400      [7.14e-04, 1.32e-04, 6.92e-05, 6.18e-05, 1.04e-04, 6.85e-05]    [6.93e-04, 1.32e-04, 6.92e-05, 6.18e-05, 1.04e-04, 6.85e-05]    [1.03e-02, 3.67e-02]    
4600      [6.74e-04, 1.26e-04, 6.67e-05, 5.94e-05, 9.90e-05, 6.56e-05]    [6.59e-04, 1.26e-04, 6.67e-05, 5.94e-05, 9.90e-05, 6.56e-05]    [1.03e-02, 3.67e-02]    
4800      [6.49e-04, 1.21e-04, 6.48e-05, 5.85e-05, 9.22e-05, 6.34e-05]    [6.32e-04, 1.21e-04, 6.48e-05, 5.85e-05, 9.22e-05, 6.34e-05]    [1.03e-02, 3.67e-02]    
5000      [8.01e-03, 1.16e-04, 6.24e-05, 4.47e-04, 1.05e-04, 2.51e-04]    [2.54e-03, 1.16e-04, 6.24e-05, 4.47e-04, 1.05e-04, 2.51e-04]    [1.01e-02, 3.63e-02]    
5200      [7.13e-04, 1.11e-04, 6.06e-05, 5.79e-05, 8.24e-05, 6.51e-05]    [5.39e-04, 1.11e-04, 6.06e-05, 5.79e-05, 8.24e-05, 6.51e-05]    [1.03e-02, 3.67e-02]    
5400      [8.33e-03, 1.09e-04, 5.82e-05, 5.97e-04, 8.29e-05, 2.90e-04]    [3.31e-03, 1.09e-04, 5.82e-05, 5.97e-04, 8.29e-05, 2.90e-04]    [1.07e-02, 3.74e-02]    
5600      [5.26e-04, 1.04e-04, 5.62e-05, 5.21e-05, 7.47e-05, 5.50e-05]    [4.86e-04, 1.04e-04, 5.62e-05, 5.21e-05, 7.47e-05, 5.50e-05]    [1.04e-02, 3.68e-02]    
5800      [5.26e-04, 9.98e-05, 5.51e-05, 5.41e-05, 7.05e-05, 5.28e-05]    [4.47e-04, 9.98e-05, 5.51e-05, 5.41e-05, 7.05e-05, 5.28e-05]    [1.04e-02, 3.69e-02]    
6000      [1.71e-03, 9.98e-05, 5.20e-05, 8.77e-05, 8.02e-05, 1.29e-04]    [1.06e-03, 9.98e-05, 5.20e-05, 8.77e-05, 8.02e-05, 1.29e-04]    [9.97e-03, 3.61e-02]    
6200      [4.62e-04, 9.37e-05, 5.14e-05, 4.85e-05, 6.32e-05, 5.06e-05]    [4.23e-04, 9.37e-05, 5.14e-05, 4.85e-05, 6.32e-05, 5.06e-05]    [1.04e-02, 3.68e-02]    
6400      [1.24e-02, 1.04e-04, 6.88e-05, 4.85e-04, 1.62e-04, 5.38e-04]    [4.89e-03, 1.04e-04, 6.88e-05, 4.85e-04, 1.62e-04, 5.38e-04]    [1.23e-02, 4.01e-02]    
6600      [4.29e-04, 8.78e-05, 4.89e-05, 4.62e-05, 5.66e-05, 4.85e-05]    [3.87e-04, 8.78e-05, 4.89e-05, 4.62e-05, 5.66e-05, 4.85e-05]    [1.04e-02, 3.68e-02]    
6800      [4.44e-04, 8.54e-05, 4.80e-05, 5.00e-05, 5.34e-05, 4.67e-05]    [3.52e-04, 8.54e-05, 4.80e-05, 5.00e-05, 5.34e-05, 4.67e-05]    [1.04e-02, 3.69e-02]    
7000      [4.45e-04, 8.20e-05, 4.68e-05, 4.90e-05, 5.06e-05, 4.75e-05]    [3.42e-04, 8.20e-05, 4.68e-05, 4.90e-05, 5.06e-05, 4.75e-05]    [1.04e-02, 3.68e-02]    
7200      [1.10e-03, 8.01e-05, 4.55e-05, 8.02e-05, 4.90e-05, 6.38e-05]    [6.57e-04, 8.01e-05, 4.55e-05, 8.02e-05, 4.90e-05, 6.38e-05]    [1.06e-02, 3.71e-02]    
7400      [3.90e-04, 7.74e-05, 4.49e-05, 4.40e-05, 4.53e-05, 4.37e-05]    [3.22e-04, 7.74e-05, 4.49e-05, 4.40e-05, 4.53e-05, 4.37e-05]    [1.04e-02, 3.68e-02]    
7600      [9.62e-04, 7.53e-05, 4.44e-05, 8.08e-05, 4.29e-05, 5.68e-05]    [4.38e-04, 7.53e-05, 4.44e-05, 8.08e-05, 4.29e-05, 5.68e-05]    [1.03e-02, 3.66e-02]    
7800      [4.29e-04, 7.39e-05, 4.34e-05, 4.04e-05, 4.16e-05, 3.93e-05]    [3.31e-04, 7.39e-05, 4.34e-05, 4.04e-05, 4.16e-05, 3.93e-05]    [1.05e-02, 3.70e-02]    
8000      [4.22e-04, 7.15e-05, 4.22e-05, 4.16e-05, 3.97e-05, 4.45e-05]    [3.48e-04, 7.15e-05, 4.22e-05, 4.16e-05, 3.97e-05, 4.45e-05]    [1.05e-02, 3.70e-02]    
8200      [3.28e-04, 6.97e-05, 4.14e-05, 3.75e-05, 3.80e-05, 4.21e-05]    [2.86e-04, 6.97e-05, 4.14e-05, 3.75e-05, 3.80e-05, 4.21e-05]    [1.04e-02, 3.69e-02]    
8400      [3.18e-04, 6.78e-05, 4.07e-05, 3.66e-05, 3.62e-05, 4.11e-05]    [2.75e-04, 6.78e-05, 4.07e-05, 3.66e-05, 3.62e-05, 4.11e-05]    [1.04e-02, 3.69e-02]    
8600      [3.30e-03, 6.64e-05, 3.93e-05, 2.54e-04, 3.58e-05, 1.37e-04]    [1.34e-03, 6.64e-05, 3.93e-05, 2.54e-04, 3.58e-05, 1.37e-04]    [1.07e-02, 3.73e-02]    
8800      [3.50e-04, 6.45e-05, 3.97e-05, 3.51e-05, 3.28e-05, 4.25e-05]    [2.78e-04, 6.45e-05, 3.97e-05, 3.51e-05, 3.28e-05, 4.25e-05]    [1.04e-02, 3.68e-02]    
9000      [3.08e-04, 6.27e-05, 3.88e-05, 3.51e-05, 3.11e-05, 3.93e-05]    [2.48e-04, 6.27e-05, 3.88e-05, 3.51e-05, 3.11e-05, 3.93e-05]    [1.04e-02, 3.69e-02]    
9200      [2.91e-04, 6.13e-05, 3.81e-05, 3.25e-05, 3.01e-05, 3.88e-05]    [2.51e-04, 6.13e-05, 3.81e-05, 3.25e-05, 3.01e-05, 3.88e-05]    [1.05e-02, 3.70e-02]    
9400      [5.16e-04, 5.99e-05, 3.81e-05, 3.75e-05, 3.09e-05, 3.43e-05]    [3.00e-04, 5.99e-05, 3.81e-05, 3.75e-05, 3.09e-05, 3.43e-05]    [1.06e-02, 3.72e-02]    
9600      [2.68e-04, 5.85e-05, 3.69e-05, 3.07e-05, 2.79e-05, 3.72e-05]    [2.30e-04, 5.85e-05, 3.69e-05, 3.07e-05, 2.79e-05, 3.72e-05]    [1.05e-02, 3.70e-02]    
9800      [2.81e-04, 5.66e-05, 3.64e-05, 3.01e-05, 2.66e-05, 3.80e-05]    [2.29e-04, 5.66e-05, 3.64e-05, 3.01e-05, 2.66e-05, 3.80e-05]    [1.05e-02, 3.70e-02]    
10000     [2.59e-04, 5.57e-05, 3.58e-05, 2.99e-05, 2.55e-05, 3.60e-05]    [2.13e-04, 5.57e-05, 3.58e-05, 2.99e-05, 2.55e-05, 3.60e-05]    [1.05e-02, 3.70e-02]    
10200     [3.35e-04, 5.45e-05, 3.52e-05, 3.16e-05, 2.46e-05, 3.92e-05]    [2.52e-04, 5.45e-05, 3.52e-05, 3.16e-05, 2.46e-05, 3.92e-05]    [1.05e-02, 3.70e-02]    

Best model at step 10000:
  train loss: 4.42e-04
  test loss: 3.96e-04
  test metric: [1.05e-02, 3.70e-02]

'train' took 170.610247 s

But when PDEPointResampler is enabled, some of the test losses grow very much. Here especially the 4th boundary condition - the Nuemann condition for $y=+\pi$. But the metrics are still the same.

Training log when PDEPointResampler is enabled

Training model...

Step      Train loss                                                      Test loss                                                       Test metric             
0         [2.47e+01, 6.13e-01, 4.66e-02, 2.29e+00, 1.27e+00, 4.02e+00]    [1.51e+01, 6.13e-01, 4.66e-02, 2.29e+00, 1.27e+00, 4.02e+00]    [4.01e+00, 7.23e-01]    
200       [1.41e+00, 4.07e-02, 3.24e-02, 1.20e-01, 1.07e-02, 4.25e-01]    [7.80e-01, 4.07e-02, 3.24e-02, 1.20e-01, 1.07e-02, 4.25e-01]    [2.71e-01, 1.88e-01]    
400       [4.45e-02, 3.35e-03, 3.68e-03, 4.82e-03, 5.70e-03, 6.48e-03]    [4.22e-02, 3.35e-03, 3.68e-03, 4.82e-03, 5.70e-03, 6.48e-03]    [1.52e-02, 4.45e-02]    
600       [1.63e-02, 2.23e-03, 1.35e-03, 2.39e-03, 1.92e-03, 3.83e-03]    [1.67e-02, 2.23e-03, 1.35e-03, 2.39e-03, 1.92e-03, 3.83e-03]    [1.23e-02, 4.00e-02]    
800       [1.00e-02, 1.60e-03, 6.82e-04, 1.55e-03, 8.09e-04, 2.26e-03]    [9.13e-03, 1.60e-03, 6.82e-04, 1.55e-03, 8.09e-04, 2.26e-03]    [1.11e-02, 3.81e-02]    
1000      [7.20e-03, 1.17e-03, 4.14e-04, 1.12e-03, 4.42e-04, 1.34e-03]    [5.96e-03, 1.17e-03, 4.14e-04, 1.12e-03, 4.42e-04, 1.34e-03]    [1.06e-02, 3.72e-02]    
1200      [5.60e-03, 9.54e-04, 2.77e-04, 8.70e-04, 3.17e-04, 8.23e-04]    [4.46e-03, 6.74e-03, 1.02e-03, 1.47e-02, 4.24e-01, 8.23e-04]    [1.03e-02, 3.67e-02]    
1400      [7.56e-03, 7.34e-04, 1.96e-04, 6.22e-04, 2.53e-04, 6.61e-04]    [4.10e-03, 6.78e-03, 9.34e-04, 1.37e-02, 4.26e-01, 6.61e-04]    [1.00e-02, 3.62e-02]    
1600      [3.86e-03, 5.80e-04, 1.57e-04, 4.54e-04, 2.36e-04, 4.43e-04]    [3.04e-03, 6.87e-03, 8.81e-04, 1.33e-02, 4.27e-01, 4.43e-04]    [1.03e-02, 3.67e-02]    
1800      [5.67e-03, 4.60e-04, 1.33e-04, 5.25e-04, 2.15e-04, 4.74e-04]    [3.02e-03, 6.96e-03, 8.47e-04, 1.27e-02, 4.28e-01, 4.74e-04]    [1.03e-02, 3.67e-02]    
2000      [2.40e-03, 3.96e-04, 1.13e-04, 2.56e-04, 1.98e-04, 2.52e-04]    [2.32e-03, 7.06e-03, 8.30e-04, 1.24e-02, 4.28e-01, 2.52e-04]    [1.01e-02, 3.63e-02]    
2200      [2.07e-03, 3.47e-04, 1.05e-04, 2.06e-04, 1.67e-04, 2.03e-04]    [2.03e-03, 7.32e-03, 8.34e-04, 1.20e-02, 3.89e-01, 2.03e-04]    [1.01e-02, 3.63e-02]    
2400      [2.09e-03, 3.02e-04, 9.96e-05, 1.71e-04, 1.61e-04, 1.75e-04]    [1.93e-03, 7.39e-03, 8.23e-04, 1.19e-02, 3.89e-01, 1.75e-04]    [9.96e-03, 3.60e-02]    
2600      [2.49e-03, 2.70e-04, 9.46e-05, 2.57e-04, 1.53e-04, 1.67e-04]    [2.26e-03, 7.43e-03, 8.09e-04, 1.22e-02, 3.89e-01, 1.67e-04]    [1.02e-02, 3.64e-02]    
2800      [1.36e-03, 2.37e-04, 9.30e-05, 1.20e-04, 1.45e-04, 1.23e-04]    [1.39e-03, 7.48e-03, 7.97e-04, 1.16e-02, 3.89e-01, 1.23e-04]    [1.02e-02, 3.65e-02]    
3000      [1.22e-03, 2.16e-04, 9.05e-05, 1.06e-04, 1.39e-04, 1.08e-04]    [1.26e-03, 7.53e-03, 7.90e-04, 1.15e-02, 3.89e-01, 1.08e-04]    [1.03e-02, 3.66e-02]    
3200      [1.43e-03, 1.82e-04, 8.62e-05, 1.25e-04, 1.38e-04, 1.20e-04]    [1.07e-03, 6.86e-03, 7.38e-04, 1.18e-02, 4.33e-01, 1.20e-04]    [1.04e-02, 3.69e-02]    
3400      [2.13e-03, 1.75e-04, 7.97e-05, 8.70e-05, 1.46e-04, 1.34e-04]    [1.47e-03, 6.88e-03, 7.31e-04, 1.19e-02, 4.32e-01, 1.34e-04]    [9.90e-03, 3.59e-02]    
3600      [9.20e-04, 1.60e-04, 7.90e-05, 7.82e-05, 1.28e-04, 8.20e-05]    [9.57e-04, 6.91e-03, 7.25e-04, 1.18e-02, 4.32e-01, 8.20e-05]    [1.03e-02, 3.67e-02]    
3800      [8.75e-04, 1.50e-04, 7.62e-05, 7.30e-05, 1.22e-04, 7.96e-05]    [8.20e-04, 6.94e-03, 7.19e-04, 1.18e-02, 4.32e-01, 7.96e-05]    [1.03e-02, 3.66e-02]    
4000      [1.04e-03, 1.44e-04, 7.27e-05, 7.67e-05, 1.19e-04, 8.96e-05]    [9.54e-04, 6.97e-03, 7.15e-04, 1.18e-02, 4.32e-01, 8.96e-05]    [1.01e-02, 3.64e-02]    
4200      [7.44e-04, 1.40e-04, 6.86e-05, 6.06e-05, 1.15e-04, 6.93e-05]    [7.57e-04, 7.44e-03, 6.97e-04, 1.17e-02, 4.23e-01, 6.93e-05]    [1.03e-02, 3.67e-02]    
4400      [6.96e-04, 1.33e-04, 6.58e-05, 5.83e-05, 1.08e-04, 6.71e-05]    [6.92e-04, 7.46e-03, 6.93e-04, 1.16e-02, 4.23e-01, 6.71e-05]    [1.04e-02, 3.67e-02]    
4600      [6.58e-04, 1.27e-04, 6.35e-05, 5.65e-05, 1.02e-04, 6.46e-05]    [6.45e-04, 7.49e-03, 6.90e-04, 1.16e-02, 4.23e-01, 6.46e-05]    [1.04e-02, 3.68e-02]    
4800      [9.13e-04, 1.20e-04, 6.27e-05, 8.23e-05, 9.34e-05, 7.33e-05]    [5.98e-04, 7.52e-03, 6.86e-04, 1.15e-02, 4.23e-01, 7.33e-05]    [1.03e-02, 3.67e-02]    
5000      [8.35e-04, 1.18e-04, 5.92e-05, 5.31e-05, 9.15e-05, 7.61e-05]    [6.85e-04, 7.52e-03, 6.82e-04, 1.16e-02, 4.23e-01, 7.61e-05]    [1.02e-02, 3.65e-02]    
5200      [6.58e-04, 1.03e-04, 5.68e-05, 6.50e-05, 8.29e-05, 6.29e-05]    [5.10e-04, 6.95e-03, 6.72e-04, 1.23e-02, 4.59e-01, 6.29e-05]    [1.03e-02, 3.67e-02]    
5400      [2.65e-03, 1.02e-04, 6.14e-05, 1.72e-04, 9.45e-05, 1.41e-04]    [1.21e-03, 6.98e-03, 6.72e-04, 1.24e-02, 4.60e-01, 1.41e-04]    [1.11e-02, 3.80e-02]    
5600      [5.26e-04, 9.78e-05, 5.33e-05, 5.39e-05, 7.29e-05, 5.80e-05]    [4.76e-04, 6.98e-03, 6.65e-04, 1.23e-02, 4.58e-01, 5.80e-05]    [1.04e-02, 3.68e-02]    
5800      [1.46e-03, 9.58e-05, 5.16e-05, 1.21e-04, 7.02e-05, 8.06e-05]    [6.03e-04, 6.98e-03, 6.59e-04, 1.23e-02, 4.59e-01, 8.06e-05]    [1.03e-02, 3.67e-02]    
6000      [4.78e-04, 9.31e-05, 5.00e-05, 5.08e-05, 6.53e-05, 5.50e-05]    [4.41e-04, 6.99e-03, 6.58e-04, 1.23e-02, 4.58e-01, 5.50e-05]    [1.04e-02, 3.68e-02]    
6200      [5.40e-04, 1.02e-04, 5.21e-05, 5.73e-05, 6.39e-05, 5.39e-05]    [4.01e-04, 7.28e-03, 6.41e-04, 1.21e-02, 4.09e-01, 5.39e-05]    [1.04e-02, 3.68e-02]    
6400      [4.60e-04, 9.71e-05, 5.11e-05, 4.72e-05, 5.99e-05, 5.19e-05]    [4.29e-04, 7.29e-03, 6.41e-04, 1.22e-02, 4.09e-01, 5.19e-05]    [1.04e-02, 3.68e-02]    
6600      [4.32e-04, 9.30e-05, 4.97e-05, 4.63e-05, 5.76e-05, 4.99e-05]    [3.95e-04, 7.29e-03, 6.38e-04, 1.21e-02, 4.09e-01, 4.99e-05]    [1.04e-02, 3.68e-02]    
6800      [4.14e-04, 8.96e-05, 4.85e-05, 4.51e-05, 5.50e-05, 4.85e-05]    [3.76e-04, 7.30e-03, 6.35e-04, 1.21e-02, 4.09e-01, 4.85e-05]    [1.04e-02, 3.69e-02]    
7000      [4.05e-04, 8.63e-05, 4.80e-05, 4.50e-05, 5.20e-05, 4.62e-05]    [3.54e-04, 7.31e-03, 6.34e-04, 1.21e-02, 4.09e-01, 4.62e-05]    [1.04e-02, 3.69e-02]    
7200      [4.24e-04, 8.35e-05, 4.66e-05, 4.33e-05, 4.64e-05, 4.74e-05]    [3.78e-04, 7.48e-03, 6.68e-04, 1.22e-02, 4.12e-01, 4.74e-05]    [1.04e-02, 3.69e-02]    
7400      [3.72e-04, 8.10e-05, 4.56e-05, 4.12e-05, 4.48e-05, 4.44e-05]    [3.32e-04, 7.49e-03, 6.66e-04, 1.22e-02, 4.12e-01, 4.44e-05]    [1.04e-02, 3.69e-02]    
7600      [9.87e-04, 7.83e-05, 4.69e-05, 7.49e-05, 4.93e-05, 5.39e-05]    [5.01e-04, 7.51e-03, 6.67e-04, 1.21e-02, 4.13e-01, 5.39e-05]    [1.07e-02, 3.74e-02]    
7800      [3.50e-04, 7.60e-05, 4.38e-05, 3.91e-05, 4.12e-05, 4.36e-05]    [3.10e-04, 7.50e-03, 6.64e-04, 1.22e-02, 4.12e-01, 4.36e-05]    [1.04e-02, 3.69e-02]    
8000      [8.20e-03, 7.44e-05, 4.52e-05, 5.89e-04, 4.44e-05, 2.32e-04]    [2.78e-03, 7.50e-03, 6.59e-04, 1.21e-02, 4.12e-01, 2.32e-04]    [1.02e-02, 3.65e-02]    
8200      [3.29e-04, 6.90e-05, 3.94e-05, 3.74e-05, 4.16e-05, 4.35e-05]    [2.81e-04, 7.60e-03, 6.53e-04, 1.15e-02, 4.21e-01, 4.35e-05]    [1.04e-02, 3.69e-02]    
8400      [3.65e-04, 6.72e-05, 3.91e-05, 4.20e-05, 3.88e-05, 4.35e-05]    [2.73e-04, 7.61e-03, 6.51e-04, 1.15e-02, 4.21e-01, 4.35e-05]    [1.04e-02, 3.69e-02]    
8600      [3.32e-04, 6.57e-05, 3.80e-05, 3.66e-05, 3.64e-05, 4.27e-05]    [2.66e-04, 7.61e-03, 6.50e-04, 1.15e-02, 4.21e-01, 4.27e-05]    [1.04e-02, 3.68e-02]    
8800      [3.07e-04, 6.42e-05, 3.75e-05, 3.32e-05, 3.48e-05, 4.11e-05]    [2.65e-04, 7.62e-03, 6.50e-04, 1.15e-02, 4.21e-01, 4.11e-05]    [1.05e-02, 3.69e-02]    
9000      [2.90e-04, 6.29e-05, 3.66e-05, 3.21e-05, 3.32e-05, 4.10e-05]    [2.56e-04, 7.61e-03, 6.48e-04, 1.15e-02, 4.21e-01, 4.10e-05]    [1.04e-02, 3.69e-02]    
9200      [2.82e-04, 6.35e-05, 3.49e-05, 3.28e-05, 2.90e-05, 3.85e-05]    [2.47e-04, 7.48e-03, 6.22e-04, 1.18e-02, 4.54e-01, 3.85e-05]    [1.05e-02, 3.70e-02]    
9400      [2.73e-04, 6.17e-05, 3.47e-05, 3.19e-05, 2.81e-05, 3.79e-05]    [2.41e-04, 7.48e-03, 6.23e-04, 1.18e-02, 4.54e-01, 3.79e-05]    [1.05e-02, 3.70e-02]    
9600      [3.37e-04, 5.98e-05, 3.40e-05, 3.48e-05, 2.72e-05, 4.29e-05]    [2.78e-04, 7.48e-03, 6.24e-04, 1.18e-02, 4.54e-01, 4.29e-05]    [1.05e-02, 3.70e-02]    
9800      [2.14e-03, 5.88e-05, 3.73e-05, 2.10e-04, 3.91e-05, 1.03e-04]    [8.29e-04, 7.49e-03, 6.23e-04, 1.16e-02, 4.56e-01, 1.03e-04]    [1.08e-02, 3.75e-02]    
10000     [8.35e-04, 5.71e-05, 3.41e-05, 6.15e-05, 2.46e-05, 5.24e-05]    [4.65e-04, 7.48e-03, 6.26e-04, 1.20e-02, 4.55e-01, 5.24e-05]    [1.06e-02, 3.72e-02]    
10200     [3.00e-04, 5.40e-05, 3.24e-05, 2.79e-05, 2.58e-05, 3.73e-05]    [2.40e-04, 7.64e-03, 6.12e-04, 1.23e-02, 4.25e-01, 3.73e-05]    [1.05e-02, 3.70e-02]    

Best model at step 9400:
  train loss: 4.67e-04
  test loss: 4.75e-01
  test metric: [1.05e-02, 3.70e-02]

'train' took 174.835359 s

I have tried to increase of decrease the training points number, change the resampling frequency - still the same: if I get convergence according to the training loss and metrics, I still get bad test losses.

I have compared the predicted and true derivative at $y=+\pi$ for the regular grid, and it looks almost the same.

The code

def get_dx(x, y):
    return dde.grad.jacobian(y, x, i=0, j=0)

def get_dy(x, y):
    return dde.grad.jacobian(y, x, i=0, j=1)

x = np.linspace(LEFT, RIGHT, 101)
t = np.linspace(0, T, 51)
y = np.array([TOP])

points = np.stack(np.meshgrid(x, y, t), axis=-1).reshape(len(x), len(t), 3)

U_Y_pred = model.predict(points.reshape(-1, points.shape[-1]), operator=get_dy).reshape(points.shape[:2])
U_Y_true = du_dy(points.reshape(-1, points.shape[-1])).reshape(points.shape[:2])

print(f'MSE loss: {np.mean((U_Y_pred - U_Y_true) ** 2):.6e}')

fig, ax = plt.subplots(1, 3, figsize=(15, 6))

def show(func, ax, title):
    im = ax.imshow(func, extent=[0, T, LEFT, RIGHT], origin='lower')
    ax.set_title(title)
    fig.colorbar(im, ax=ax)

show(U_Y_pred, ax[0], f'PINN Derivative $u_y$ (y={TOP})')
show(U_Y_true, ax[1], f'True Derivative $u_y$ (y={TOP})')
show(np.abs(U_Y_pred - U_Y_true), ax[2], 'Difference |Pred - True|')

for a in ax:
    a.set_xlabel('t')
    a.set_ylabel('x')

plt.tight_layout()
plt.show()

The output

MSE loss: 2.638281e-05

I have also calculated this loss term for points from the training and the testing set of, as far as I understand, the last model state. I have tried to take the points from 2 different places . The losses are low.

The code and the output

The code:

TROUBLESOME_BC_IX = 3
bc = data.bcs[TROUBLESOME_BC_IX]

def calculate_loss_for_filtered_points(X):
    model.net.eval()
    X = bc.collocation_points(X)
    inputs = torch.tensor(X, requires_grad=True)
    outputs = model.net(inputs)
    error = bc.error(X, inputs, outputs, 0, len(inputs))
    print('Filtered points number:', len(X))
    print(f'MSE: {torch.mean(error ** 2).item():.6e}')
    print()


print('data.test_x')
calculate_loss_for_filtered_points(data.test_x)

print('train_state.X_test')
calculate_loss_for_filtered_points(train_state.X_test)

print('data.train_x')
calculate_loss_for_filtered_points(data.train_x)

print('train_state.X_train')
calculate_loss_for_filtered_points(train_state.X_train)

The output:

data.test_x
Filtered points number: 834
MSE: 2.566222e-05

train_state.X_test
Filtered points number: 834
MSE: 2.566222e-05

data.train_x
Filtered points number: 1668
MSE: 2.580953e-05

train_state.X_train
Filtered points number: 1668
MSE: 2.580953e-05

Another way to calculate the loss: with model._outputs_losses(). The results is the same as in the last entry of the training log.

The code and the output

The code:

outputs, losses = model._outputs_losses(
    training=False,
    inputs=train_state.X_test,
    targets=train_state.y_test,
    auxiliary_vars=train_state.test_aux_vars
)
print(f'Losses from `model._outputs_losses` for `train_state.X_test`: {losses}')

The output:

Losses from `model._outputs_losses` for `train_state.X_test`: [2.4006059e-04 7.6403474e-03 6.1155070e-04 1.2318139e-02 4.2543173e-01
 3.7326652e-05]

I was analysing this behavior with Gemini, DeepSeek and Qwen Code, and the latter told me to check the BC error calculation cycle in PDE.losses() in deepxde/data/pde.py. Indeed, it seems that for calculation of test losses train points are sent to bc.error() along with actual inputs and outputs. There are beg and end pointers which must select the correct span from the train points array, but probably after resampling something breaks?

I have created a copy of NeumannBC class, and added some debug output there:

• shapes of X and inputs; values of beg and end
• 10 first and 10 last elements of X and input after [beg : end] elements were chosen (i.e. filtered)
• min and max values for each coordinate column of X and input elements were chosen
• plots of column-wise difference of X and input (filtered) and of difference between func(X_filtered) and func(inputs_filtered)

As far as I understand, the filtered inputs and filtered X must be the same, as they are used to calculate derivative values at boundary points and normal values at these points. But the difference is non-zero.

The code

class NeumannBC(dde.icbc.BC):
    """Neumann boundary conditions: dy/dn(x) = func(x)."""

    def __init__(self, geom, func, on_boundary, component=0):
        super().__init__(geom, on_boundary, component)
        self.func = dde.icbc.boundary_conditions.npfunc_range_autocache(dde.utils.return_tensor(func))

    def error(self, X, inputs, outputs, beg, end, aux_var=None):
        print(f'{X.shape=}')
        print(f'{inputs.shape=}')
        print(f'{outputs.shape=}')
        print(f'{beg=}, {end=}')
        print()
        print('10 first elements of `X[beg : end]`:', X[beg : end][:10])
        print('10 first elements of `inputs[beg : end]`:', inputs[beg : end][:10])
        print()
        print('10 last elements of `X[beg : end]`:', X[beg : end][-10:])
        print('10 last elements of `inputs[beg : end]`:', inputs[beg : end][-10:])
        print()

        def print_stats(x):
            if isinstance(x, torch.Tensor):
                x = x.detach().cpu().numpy()
            print('Min and Max of the 0-th coordinate:', x[:, 0].min(), x[:, 0].max())
            print('Min and Max of the 1-th coordinate:', x[:, 1].min(), x[:, 1].max())
            print('Min and Max of the 2-th coordinate:', x[:, 2].min(), x[:, 2].max())
            print()

        print('Stats for `X[beg : end]`')
        print_stats(X[beg : end])

        print('Stats for `inputs[beg : end]`')
        print_stats(inputs[beg : end])
        
        print('Are `X[beg : end]` and `inputs[beg : end]` all close:', np.allclose(X[beg : end], inputs[beg : end].detach().cpu().numpy()))

        a = X[beg : end]
        b = inputs[beg : end].detach().cpu().numpy()
        plt.plot(np.abs(a - b)[:, 0], label='0-th coordinate')
        plt.plot(np.abs(a - b)[:, 1], label='1-th coordinate')
        plt.plot(np.abs(a - b)[:, 2], label='2-th coordinate')
        plt.title('abs(X[beg : end] - inputs[beg : end])')
        plt.legend()
        plt.show()
        
        values = self.func(X, beg, end, aux_var)
        
        values_ = self.func(inputs.detach().cpu().numpy(), beg, end, aux_var)
        plt.plot(torch.abs(values - values_).detach().cpu().numpy())
        plt.title('abs(func(X, ...) - func(inputs, ...))')
        plt.show()
        
        return self.normal_derivative(X, inputs, outputs, beg, end) - values


bc_ = NeumannBC(geomtime, du_dy, on_boundary_top)

# Just like inside PDE.losses()
SEND_AS_X = data.train_x
SEND_AS_INPUTS = data.test_x

bcs_start = np.cumsum([0] + data.num_bcs)
inputs = torch.tensor(SEND_AS_INPUTS, requires_grad=True)
outputs = model.net(inputs)
beg = bcs_start[TROUBLESOME_BC_IX]
end = bcs_start[TROUBLESOME_BC_IX + 1]
error = bc_.error(SEND_AS_X, inputs, outputs, beg, end)
print('`data.train_x` as `X`, `data.test_x` as `inputs`')
print(f'{beg=}, {end=}')
print(f'MSE loss: {torch.mean(error ** 2).item():.6e}')

The output

X.shape=(24999, 3)
inputs.shape=torch.Size([11223, 3])
outputs.shape=torch.Size([11223, 1])
beg=np.int64(4165), end=np.int64(4999)

10 first elements of `X[beg : end]`: [[0.7853985  3.1415927  0.96203613]
 [1.9634962  3.1415927  0.93566895]
 [2.5525446  3.1415927  0.00708008]
 [0.1963501  3.1415927  0.2397461 ]
 [1.3744469  3.1415927  0.72436523]
 [2.8470688  3.1415927  0.00830078]
 [0.4908743  3.1415927  0.8996582 ]
 [1.6689711  3.1415927  0.33911133]
 [2.2580204  3.1415927  0.6378174 ]
 [1.0799227  3.1415927  0.3466797 ]]
10 first elements of `inputs[beg : end]`: tensor([[0.7854, 3.1416, 0.7432],
        [1.9635, 3.1416, 0.6602],
        [2.5525, 3.1416, 0.1543],
        [0.1964, 3.1416, 0.0195],
        [1.3744, 3.1416, 0.0831],
        [2.8471, 3.1416, 0.8927],
        [0.4909, 3.1416, 0.2634],
        [1.6690, 3.1416, 0.1736],
        [2.2580, 3.1416, 0.6191],
        [1.0799, 3.1416, 0.4072]], device='cuda:0', grad_fn=<SliceBackward0>)

10 last elements of `X[beg : end]`: [[1.758709   3.1415927  0.15283203]
 [2.3477583  3.1415927  0.69262695]
 [1.1696606  3.1415927  0.96691895]
 [2.6422825  3.1415927  0.44543457]
 [0.286088   3.1415927  0.6191406 ]
 [1.4641848  3.1415927  0.34191895]
 [2.0532331  3.1415927  0.17578125]
 [0.8751364  3.1415927  0.66137695]
 [3.010438   3.1415927  0.1784668 ]
 [0.65424347 3.1415927  0.9194336 ]]
10 first elements of `inputs[beg : end]`: tensor([[1.7587, 3.1416, 0.3684],
        [2.3478, 3.1416, 0.9021],
        [1.1697, 3.1416, 0.2833],
        [2.6423, 3.1416, 0.1138],
        [0.2861, 3.1416, 0.0977],
        [1.4642, 3.1416, 0.0050],
        [2.0532, 3.1416, 0.6089],
        [0.8751, 3.1416, 0.3447],
        [3.0104, 3.1416, 0.4083],
        [0.6542, 3.1416, 0.5125]], device='cuda:0', grad_fn=<SliceBackward0>)

Stats for `X[beg : end]`
Min and Max of the 0-th coordinate: 0.0030679703 3.1392918
Min and Max of the 1-th coordinate: 3.1415927 3.1415927
Min and Max of the 2-th coordinate: 0.00012207031 0.9995117

Stats for `inputs[beg : end]`
Min and Max of the 0-th coordinate: 0.0030679703 3.1392918
Min and Max of the 1-th coordinate: 3.1415927 3.1415927
Min and Max of the 2-th coordinate: 0.0010986328 0.9980469

Are `X[beg : end]` and `inputs[beg : end]` all close: False

If I set SEND_AS_X and SEND_AS_INPUTS to be the same (either data.test_x or data.train_x), or if I disable the PDEPointResampler, then the difference goes away.

The output

Are `X[beg : end]` and `inputs[beg : end]` all close: True

Am I doing anything wrong, or can there be a bug inside DeepXDE resampling or loss calculation logic?

The whole Jupyter Notebook

[echen5503]

Given what you're reporting, this is likely a bug on DeepXDE's end, but I will have to take a closer look. DeepXDE is known to be a bit buggy with its domain geometry methods. Thank you for your detailed report, we will get back to you soon.

[kyouma]

I have found out the source of this error.

When the training points are resampled, the testing points for boundary conditions (BC) are still linked to the old training BC points. The training and testing BC losses are calculated in deepxde/data/pde.py in losses():

error = bc.error(self.train_x, inputs, outputs, beg, end)

Even if the number of training BC points inside the new, resampled self.train_x are the same as before, i.e. if the beg and end that come from self.num_bcs are still correct, then still the BC points in self.train_x after the resampling differ from the BC points in inputs.

I am not sure if this happens for IDE, FPDE, TimeFPDE and PDEOperator.

One possible quick solution may be right before the bc.error() call:

1. filter inputs with bc.collocation_points()
2. set beg to 0 and end to len(inputs)
3. replace self.train_x with inputs.detach().cpu().numpy() - for the PyTorch backend - and make a function in deepxde.util to convert a tensor to a NumPy array for each backend.

It will create an overhead, which may be removed for the training step by passing also a flag is_training or something. And I'm not sure how this beg and end may influence the inner logic.

Another solution, I think, is to rewrite the testing points sampling logic:

1. sample and store testing BC points independently from the training BC points
2. store num_bcs for testing points, too
3. pass the points and their num_bcs to losses() via Data.losses_train() and Data.losses_test().

I have already done this on my PC, now I am testing and comparing the sampling behavior. I may try to make a pool request.

I see that for the inner domain points, the testing sample gets a uniform grid. But for BC it supports pseudorandom and low-discrepancy samplers (Hammersley by default). So for BCs the spatial sampling work the same: I get the same points. As for the time coordinates, there is a permutation operation in random_initial_points(), so I get the same spatial coordinates with permuted time coordinates - i.e. different points. I think, for the spatial coordinates one may want to tune the sampling procedure so that, e.g. Hammersly, for the training and the testing samples start generating different sequences.

As for the inner domain points, does DeepXDE sticks to a uniform grid, or is it possible to use "random" sequences, too?

[kyouma]

@echen5503 Hello.
@lululxvi Hello. I apologize for bothering you by calling you here directly.

I have made a commit, where I have:

• Added independent sampling of BC points for the test sample and tracking of its num_bcs (i.e. test_num_bcs).
  I have used uniform because the sampling strategy for the inner domain points for the test sample is uniform, and the default sampling strategy for training is Hammersley, and in other cases people may prefer pseudorandom or quasirandom for the training sample. Otherwise I can add test_distribution, just like there is train_distribution already.
  I haven't renamed num_bcs to train_num_bcs.
• Added the ability to pass the correct tensor of points and num_bcs to the PDE/TimePDE loss calculation function losses().
• Made the Data class (which is the parent of PDE and TimePDE) send the correct tensor and num_bcs to the losses() method.
• Updated the losses() method signature in PDEOperator, because it's also called from Data, and will receive the tensor of points and num_bcs. I don't know if PDEOperator suffers from this BC resampling bug, so this method may ignore these new arguments now.

The 12 jobs of building finished successfully.

If the main concept is OK, I can finalize the changes and test it as you tell me, and then make a pull request.

[kyouma]

Warning: There is a mistake in my code: in the PDE declaration the sine term in

f = torch.exp(-1.3* t) * (77/250 * torch.sin(0.6 * x) + 3807/400 * torch.cos(2.5 * y))

must have negative coefficient (-77/250 instead of 77/250):

f = torch.exp(-1.3 * t) * (-77/250 * torch.sin(0.6 * x) + 3807/400 * torch.cos(2.5 * y))

However, this is not the source of the loss difference, and the changes to the test loss computation process still must be done.

By the way, during the experiments with this problem (but an inverse version) I have probably found another bug: #2074.

[echen5503]

The changes look interesting, but we must be careful about backwards compatibility. Please open up a PR for this and we can discuss this more. I think the first step for testing would be to create a minimal code file that reproduces the issue.