timesead.models.layers ====================== .. py:module:: timesead.models.layers Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/timesead/models/layers/anom_attention/index /autoapi/timesead/models/layers/autocorrelation/index /autoapi/timesead/models/layers/autoformer_encdec/index /autoapi/timesead/models/layers/causal_conv/index /autoapi/timesead/models/layers/conv_block/index /autoapi/timesead/models/layers/conv_lstm/index /autoapi/timesead/models/layers/embed/index /autoapi/timesead/models/layers/fourier_correlation/index /autoapi/timesead/models/layers/inception/index /autoapi/timesead/models/layers/kervolution/index /autoapi/timesead/models/layers/multi_wavelet_correlation/index /autoapi/timesead/models/layers/planar_nf/index /autoapi/timesead/models/layers/same_pad/index Classes ------- .. autoapisummary:: timesead.models.layers.CausalConv1d timesead.models.layers.ConvLSTM timesead.models.layers.ConvLSTMCell timesead.models.layers.Kernel timesead.models.layers.LinearKernel timesead.models.layers.PolynomialKernel timesead.models.layers.RBFKernel timesead.models.layers.Kerv1d timesead.models.layers.PlanarFlow timesead.models.layers.PlanarTransform timesead.models.layers.SameZeroPad1d timesead.models.layers.SameCausalZeroPad1d timesead.models.layers.SameZeroPad2d timesead.models.layers.AnomalyAttention timesead.models.layers.AttentionLayer timesead.models.layers.DataEmbedding timesead.models.layers.ConvBlock timesead.models.layers.AutoCorrelationLayer timesead.models.layers.AutoCorrelation timesead.models.layers.FourierBlock timesead.models.layers.FourierCrossAttention timesead.models.layers.MultiWaveletCross timesead.models.layers.MultiWaveletTransform Functions --------- .. autoapisummary:: timesead.models.layers.calc_causal_same_pad timesead.models.layers.calc_same_pad Package Contents ---------------- .. py:class:: CausalConv1d(in_channels: int, out_channels: int, kernel_size: torch.nn.common_types._size_1_t, stride: torch.nn.common_types._size_1_t = 1, dilation: torch.nn.common_types._size_1_t = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros', device=None, dtype=None) Bases: :py:obj:`torch.nn.Conv1d` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: causal_padding .. py:method:: forward(input: torch.Tensor) -> torch.Tensor .. py:class:: ConvLSTM(*args, **kwargs) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: lstm .. py:method:: forward(x, hidden=None, memory=None) input shape: (T, B, C, H, W) .. py:class:: ConvLSTMCell(in_channels: int, hid_channels: int, kernel_size: Union[int, Tuple[int, int]], spatial_size: Tuple[int, int]) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: hid_channels .. py:attribute:: x2h .. py:attribute:: h2h .. py:attribute:: c2c .. py:method:: reset_parameters() .. py:method:: forward(x, h, c) .. py:class:: Kernel(learnable_parameters: bool = False) Bases: :py:obj:`torch.nn.Module`, :py:obj:`abc.ABC` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: learnable_parameters :value: False .. py:method:: forward(x1: torch.Tensor, x2: torch.Tensor, diag_only: bool = False) -> torch.Tensor Compute the kernel function for inputs x1 and x2 :param x1: A tensor of shape ([B1], D) :param x2: A tensor of shape ([B2], D) :param diag_only: Whether the entire kernel matrix should be computed or only the diagonal :return: A tensor of shape ([B1] + [B2]) if diag_only = False, else a tensor of shape ([B]), where [B] is the result of broadcasting [B1] and [B2]. .. py:class:: LinearKernel(learnable_parameters: bool = False) Bases: :py:obj:`Kernel` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:class:: PolynomialKernel(degree: int = 2, c0: float = 0.0, learnable_parameters: bool = False) Bases: :py:obj:`Kernel` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: degree :value: 2 .. py:attribute:: c0 :value: 0.0 .. py:class:: RBFKernel(gamma: float = 1.0, learnable_parameters: bool = False) Bases: :py:obj:`Kernel` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: gamma :value: 1.0 .. py:class:: Kerv1d(in_channels: int, out_channels: int, kernel_size: int, stride: int = 1, padding: int = 0, dilation: int = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros', kernel: Union[str, Kernel] = 'linear', learnable_kernel: bool = False) Bases: :py:obj:`torch.nn.Conv1d` Applies a 1D kervolution over an input signal composed of several inputplanes. :param in_channels: Number of channels in the input image :type in_channels: int :param out_channels: Number of channels produced by the convolution :type out_channels: int :param kernel_size: Size of the convolving kernel :type kernel_size: int or tuple :param stride: Stride of the convolution. Default: 1 :type stride: int or tuple, optional :param padding: Zero-padding added to both sides of the input. Default: 0 :type padding: int or tuple, optional :param padding_mode: `zeros` :type padding_mode: string, optional :param dilation: Spacing between kernel elements. Default: 1 :type dilation: int or tuple, optional :param groups: Number of blocked connections from input channels to output channels. Default: 1 :type groups: int, optional :param bias: If ``True``, adds a learnable bias to the output. Default: ``True`` :type bias: bool, optional :param kernel: 'linear' :type kernel: str or Kernel :param learnable_kernel: Learnable kernel parameters. Default: False :type learnable_kernel: bool Shape: - Input: :math:`(N, C_{in}, L_{in})` - Output: :math:`(N, C_{out}, L_{out})` where .. math:: L_{out} = \left\lfloor\frac{L_{in} + 2 \times \text{padding} - \text{dilation} \times (\text{kernel_size} - 1) - 1}{\text{stride}} + 1\right\rfloor .. rubric:: Examples >>> m = Kerv1d(16, 33, 3, kernel='rbf') >>> input = torch.randn(20, 16, 70) >>> output = m(input) Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: unfold .. py:attribute:: kernel :value: 'linear' .. py:method:: forward(x: torch.Tensor) -> torch.Tensor .. py:class:: PlanarFlow(dim: int, num_layers: int = 6) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Make a planar flow by stacking planar transformations in sequence. :param dim: Dimensionality of the distribution to be estimated. :param num_layers: Number of transformations in the flow. .. py:attribute:: layers .. py:method:: forward(z: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor] .. py:class:: PlanarTransform(dim: int, epsilon: float = 0.0001) Bases: :py:obj:`torch.nn.Module` Implementation of the invertible transformation used in planar flow f(z) = z + u * h(dot(w.T, z) + b) See Section 4.1 in https://arxiv.org/pdf/1505.05770.pdf. Initialise weights and bias. :param dim: Dimensionality of the distribution to be estimated. .. py:attribute:: epsilon :value: 0.0001 .. py:attribute:: w .. py:attribute:: b .. py:attribute:: u .. py:method:: forward(z: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor] .. py:method:: get_u_hat() -> None Enforce w^T u >= -1. When using h(.) = tanh(.), this is a sufficient condition for invertibility of the transformation f(z). See Appendix A.1. .. py:function:: calc_causal_same_pad(kernel_size: int, stride: int = 1, in_shape: int = 1, dilation: int = 1) -> int .. py:function:: calc_same_pad(kernel_size: int, stride: int = 1, in_shape: int = 1, dilation: int = 1) -> Tuple[int, int] .. py:class:: SameZeroPad1d(kernel_size: int, stride: int = 1, in_shape: int = 1, dilation: int = 1) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Replicates the "SAME" pad algorithm from Tensorflow. Note that Tensorflow will always assume stride = 1, whereas this implementation also takes different strides into account. :param kernel_size: Kernel size that will be used :param stride: Stride that will be used :param in_shape: Size of the input. This is only needed if stride != 1 :param dilation: Dilation that will be used .. py:attribute:: padding .. py:method:: forward(x: torch.Tensor) -> torch.Tensor .. py:class:: SameCausalZeroPad1d(kernel_size: int, stride: int = 1, in_shape: int = 1, dilation: int = 1) Bases: :py:obj:`SameZeroPad1d` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Replicates the "causal" pad algorithm from Tensorflow. Note that Tensorflow will always assume stride = 1, whereas this implementation also takes different strides into account. :param kernel_size: Kernel size that will be used :param stride: Stride that will be used :param in_shape: Size of the input. This is only needed if stride != 1 :param dilation: Dilation that will be used .. py:attribute:: padding .. py:class:: SameZeroPad2d(kernel_size: torch.nn.common_types._size_2_t, stride: torch.nn.common_types._size_2_t = 1, in_shape: torch.nn.common_types._size_2_t = 1, dilation: torch.nn.common_types._size_2_t = 1) Bases: :py:obj:`torch.nn.ZeroPad2d` Pads the input tensor boundaries with zero. For `N`-dimensional padding, use :func:`torch.nn.functional.pad()`. :param padding: the size of the padding. If is `int`, uses the same padding in all boundaries. If a 4-`tuple`, uses (:math:`\text{padding\_left}`, :math:`\text{padding\_right}`, :math:`\text{padding\_top}`, :math:`\text{padding\_bottom}`) :type padding: int, tuple Shape: - Input: :math:`(N, C, H_{in}, W_{in})` or :math:`(C, H_{in}, W_{in})`. - Output: :math:`(N, C, H_{out}, W_{out})` or :math:`(C, H_{out}, W_{out})`, where :math:`H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}` :math:`W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}` Examples:: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn.ZeroPad2d(2) >>> input = torch.randn(1, 1, 3, 3) >>> input tensor([[[[-0.1678, -0.4418, 1.9466], [ 0.9604, -0.4219, -0.5241], [-0.9162, -0.5436, -0.6446]]]]) >>> m(input) tensor([[[[ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, -0.1678, -0.4418, 1.9466, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.9604, -0.4219, -0.5241, 0.0000, 0.0000], [ 0.0000, 0.0000, -0.9162, -0.5436, -0.6446, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]]]]) >>> # using different paddings for different sides >>> m = nn.ZeroPad2d((1, 1, 2, 0)) >>> m(input) tensor([[[[ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, -0.1678, -0.4418, 1.9466, 0.0000], [ 0.0000, 0.9604, -0.4219, -0.5241, 0.0000], [ 0.0000, -0.9162, -0.5436, -0.6446, 0.0000]]]]) Replicates the "SAME" pad algorithm from Tensorflow. Note that Tensorflow will always assume stride = 1, whereas this implementation also takes different strides into account. :param kernel_size: Kernel size that will be used :param stride: Stride that will be used :param in_shape: Size of the input. This is only needed if stride != 1 :param dilation: Dilation that will be used .. py:class:: AnomalyAttention(win_size: int, mask_flag: bool = True, scale: Optional[float] = None, attention_dropout: float = 0.0, output_attention: bool = False) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: scale :value: None .. py:attribute:: mask_flag :value: True .. py:attribute:: output_attention :value: False .. py:attribute:: dropout .. py:method:: forward(queries, keys, values, sigma, attn_mask) .. py:class:: AttentionLayer(attention: torch.nn.Module, d_model: int, n_heads: int, d_keys: Optional[int] = None, d_values: Optional[int] = None) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: norm .. py:attribute:: inner_attention .. py:attribute:: query_projection .. py:attribute:: key_projection .. py:attribute:: value_projection .. py:attribute:: sigma_projection .. py:attribute:: out_projection .. py:attribute:: n_heads .. py:method:: forward(queries, keys, values, attn_mask) .. py:class:: DataEmbedding(c_in: int, d_model: int, dropout: float = 0.0, use_pos: bool = True) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: value_embedding .. py:attribute:: position_embedding .. py:attribute:: dropout .. py:method:: forward(x) .. py:class:: ConvBlock(conv_layer, out_channels: int, activation, batch_norm: bool = False) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: conv .. py:attribute:: activation .. py:attribute:: norm .. py:method:: forward(x: torch.Tensor, *args, **kwargs) -> torch.Tensor .. py:class:: AutoCorrelationLayer(correlation, d_model, n_heads, d_keys=None, d_values=None) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: inner_correlation .. py:attribute:: query_projection .. py:attribute:: key_projection .. py:attribute:: value_projection .. py:attribute:: out_projection .. py:attribute:: n_heads .. py:method:: forward(queries, keys, values, attn_mask) .. py:class:: AutoCorrelation(factor=1, scale=None, attention_dropout=0.1, output_attention=False) Bases: :py:obj:`torch.nn.Module` AutoCorrelation Mechanism with the following two phases: (1) period-based dependencies discovery (2) time delay aggregation This block can replace the self-attention family mechanism seamlessly. Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: factor :value: 1 .. py:attribute:: scale :value: None .. py:attribute:: output_attention :value: False .. py:attribute:: dropout .. py:method:: time_delay_agg_training(values, corr) SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the training phase. .. py:method:: time_delay_agg_inference(values, corr) SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the inference phase. .. py:method:: time_delay_agg_full(values, corr) Standard version of Autocorrelation .. py:method:: forward(queries, keys, values, attn_mask) .. py:class:: FourierBlock(in_channels, out_channels, seq_len, num_heads=8, modes=0, mode_select_method='random') Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: index .. py:attribute:: scale .. py:attribute:: weights1 .. py:attribute:: weights2 .. py:method:: compl_mul1d(order, x, weights) .. py:method:: forward(q, k, v, mask) .. py:class:: FourierCrossAttention(in_channels, out_channels, seq_len_q, seq_len_kv, modes=64, mode_select_method='random', activation='tanh', policy=0, num_heads=8) Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: activation :value: 'tanh' .. py:attribute:: in_channels .. py:attribute:: out_channels .. py:attribute:: index_q .. py:attribute:: index_kv .. py:attribute:: scale .. py:attribute:: weights1 .. py:attribute:: weights2 .. py:method:: compl_mul1d(order, x, weights) .. py:method:: forward(q, k, v, mask) .. py:class:: MultiWaveletCross(in_channels, out_channels, seq_len_q, seq_len_kv, modes, c=64, k=8, ich=512, L=0, base='legendre', mode_select_method='random', initializer=None, activation='tanh', **kwargs) Bases: :py:obj:`torch.nn.Module` 1D Multiwavelet Cross Attention layer. Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: c :value: 64 .. py:attribute:: k :value: 8 .. py:attribute:: L :value: 0 .. py:attribute:: max_item :value: 3 .. py:attribute:: attn1 .. py:attribute:: attn2 .. py:attribute:: attn3 .. py:attribute:: attn4 .. py:attribute:: T0 .. py:attribute:: Lk .. py:attribute:: Lq .. py:attribute:: Lv .. py:attribute:: out .. py:attribute:: modes1 .. py:method:: forward(q, k, v, mask=None) .. py:method:: wavelet_transform(x) .. py:method:: evenOdd(x) .. py:class:: MultiWaveletTransform(ich=1, k=8, alpha=16, c=128, nCZ=1, L=0, base='legendre', attention_dropout=0.1) Bases: :py:obj:`torch.nn.Module` 1D multiwavelet block. Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: k :value: 8 .. py:attribute:: c :value: 128 .. py:attribute:: L :value: 0 .. py:attribute:: nCZ :value: 1 .. py:attribute:: Lk0 .. py:attribute:: Lk1 .. py:attribute:: ich :value: 1 .. py:attribute:: MWT_CZ .. py:method:: forward(queries, keys, values, attn_mask)