EduNLP.ModelZoo

rnn

class EduNLP.ModelZoo.rnn.LM(rnn_type: str, vocab_size: int, embedding_dim: int, hidden_size: int, num_layers=1, bidirectional=False, embedding=None, model_params=None, **kwargs)

Parameters

• rnn_type：str – Legal types including RNN, LSTM, GRU, BiLSTM

• vocab_size (int) –

• embedding_dim (int) –

• hidden_size (int) –

• num_layers –

• bidirectional –

• embedding –

• model_params –

• kwargs –

Examples

>>> import torch
>>> seq_idx = torch.LongTensor([[1, 2, 3], [1, 2, 0], [3, 0, 0]])
>>> seq_len = torch.LongTensor([3, 2, 1])
>>> lm = LM("RNN", 4, 3, 2)
>>> output, hn = lm(seq_idx, seq_len)
>>> output.shape
torch.Size([3, 3, 2])
>>> hn.shape
torch.Size([1, 3, 2])
>>> lm = LM("RNN", 4, 3, 2, num_layers=2)
>>> output, hn = lm(seq_idx, seq_len)
>>> output.shape
torch.Size([3, 3, 2])
>>> hn.shape
torch.Size([2, 3, 2])

forward(seq_idx, seq_len)

Parameters

• seq_idx (Tensor) – a list of indices

• seq_len (Tensor) – length

Returns

a PackedSequence object

Return type

sequence

training: bool

class EduNLP.ModelZoo.rnn.ElmoLM(vocab_size: int, embedding_dim: int, hidden_size: int, dropout_rate: float = 0.5, batch_first=True)

training: bool

forward(seq_idx, seq_len)

Parameters

• seq_idx (Tensor, of shape (batch_size, sequence_length)) – a list of indices

• seq_len (Tensor, of shape (batch_size)) – length

Returns

• pred_forward (of shape (batch_size, sequence_length))

• pred_backward (of shape (batch_size, sequence_length))

• forward_output (of shape (batch_size, sequence_length, hidden_size))

• backward_output (of shape (batch_size, sequence_length, hidden_size))

disenqnet

class EduNLP.ModelZoo.disenqnet.DisenQNet(vocab_size, concept_size, hidden_dim, dropout, pos_weight, w_cp, w_mi, w_dis, wv=None, device='cpu')

DisenQNet training and evaluation model

Parameters

• vocab_size (int) – size of vocabulary

• concept_size (int) – number of concept classes

• hidden_dim (int) – size of word and question embedding

• dropout (float) – dropout rate

• pos_weight (float) – positive sample weight in unbalanced multi-label concept classifier

• w_cp (float) – weight of concept loss

• w_mi (float) – weight of mutual information loss

• w_dis (float) – weight of disentangling loss

• wv (torch.Tensor) – Tensor of (vocab_size, hidden_dim) or None, initial word embedding, default = None

• device (str, defaults as 'cpu') – Set device for model, examples ‘cpu’、’cuda’、’cuda:0,2’

train(train_data, test_data, epoch, lr, step_size, gamma, warm_up, n_adversarial, silent)

train DisenQNet

Parameters

• train_data – train dataloader, contains text, length, concept - text: Tensor of (batch_size, seq_len) - length: Tensor of (batch_size) - concept: Tensor of (batch_size, class_size)

• test_data – test dataloader

• epoch (int) – number of epoch

• lr (float) – initial learning rate

• step_size (int) – step_size for StepLR, period of learning rate decay

• gamma (float) – gamma for StepLR, multiplicative factor of learning rate decay

• warm_up (int) – number of epoch for warming up, without adversarial process for dis_loss

• n_adversarial (int) – ratio of disc/enc training for adversarial process

• silent (bool) – whether to log loss

inference(items: dict)

DisenQNet for i2v inference. Now not support for batch !

Parameters

• items (dict) – which contains content_idx and content_len - content_idx: Tensor of (batch_size, seq_len) - content_len: Tensor of (batch_size)

• device (str) – cpu or cuda

Returns

• embed (torch.Tensor) – Tensor of (batch_size, seq_len, hidden_dim)

• k_hidden (torch.Tensor) – Tensor of (batch_size, hidden_dim)

• i_hidden (torch.Tensor) – Tensor of (batch_size, hidden_dim)

eval(test_data)

eval DisenQNet

Parameters

• test_data – iterable, train dataset, contains text, length, concept - text: Tensor of (batch_size, seq_len) - length: Tensor of (batch_size) - concept: Tensor of (batch_size, class_size)

• device (str) – cpu or cuda

Returns

loss – average loss for test dataset

Return type

float

save_pretrained(output_dir)

load(filepath)

to(device)

set_mode(train)

save_config(config_path)

classmethod from_config(config_path)

classmethod from_pretrained(model_dir)

quesnet

class EduNLP.ModelZoo.quesnet.quesnet.FeatureExtractor(feat_size=512)

make_batch(data, device, pretrain=False)

Make batch from input data (python data / np arrays -> tensors)

load_emb(emb)

pretrain_loss(batch)

Returns pretraining loss on a batch of data

forward(*input)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class EduNLP.ModelZoo.quesnet.quesnet.ImageAE(emb_size)

encoder(item, detach_tensor=False)

decoder(emb, detach_tensor=False)

training: bool

class EduNLP.ModelZoo.quesnet.quesnet.MetaAE(meta_size, emb_size)

training: bool

class EduNLP.ModelZoo.quesnet.quesnet.SeqBatch(seqs, dtype=None, device=None)

packed()

padded(max_len=None, batch_first=False)

index(item)

invert(batch, dim=0)

class EduNLP.ModelZoo.quesnet.quesnet.Path(*args, **kwargs)

PurePath subclass that can make system calls.

Path represents a filesystem path but unlike PurePath, also offers methods to do system calls on path objects. Depending on your system, instantiating a Path will return either a PosixPath or a WindowsPath object. You can also instantiate a PosixPath or WindowsPath directly, but cannot instantiate a WindowsPath on a POSIX system or vice versa.

classmethod cwd()

Return a new path pointing to the current working directory (as returned by os.getcwd()).

classmethod home()

Return a new path pointing to the user’s home directory (as returned by os.path.expanduser(‘~’)).

samefile(other_path)

Return whether other_path is the same or not as this file (as returned by os.path.samefile()).

iterdir()

Iterate over the files in this directory. Does not yield any result for the special paths ‘.’ and ‘..’.

glob(pattern)

Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.

rglob(pattern)

Recursively yield all existing files (of any kind, including directories) matching the given relative pattern, anywhere in this subtree.

absolute()

Return an absolute version of this path. This function works even if the path doesn’t point to anything.

No normalization is done, i.e. all ‘.’ and ‘..’ will be kept along. Use resolve() to get the canonical path to a file.

resolve(strict=False)

Make the path absolute, resolving all symlinks on the way and also normalizing it (for example turning slashes into backslashes under Windows).

stat()

Return the result of the stat() system call on this path, like os.stat() does.

owner()

Return the login name of the file owner.

group()

Return the group name of the file gid.

open(mode='r', buffering=- 1, encoding=None, errors=None, newline=None)

Open the file pointed by this path and return a file object, as the built-in open() function does.

read_bytes()

Open the file in bytes mode, read it, and close the file.

read_text(encoding=None, errors=None)

Open the file in text mode, read it, and close the file.

write_bytes(data)

Open the file in bytes mode, write to it, and close the file.

write_text(data, encoding=None, errors=None)

Open the file in text mode, write to it, and close the file.

touch(mode=438, exist_ok=True)

Create this file with the given access mode, if it doesn’t exist.

mkdir(mode=511, parents=False, exist_ok=False)

Create a new directory at this given path.

chmod(mode)

Change the permissions of the path, like os.chmod().

lchmod(mode)

Like chmod(), except if the path points to a symlink, the symlink’s permissions are changed, rather than its target’s.

unlink()

Remove this file or link. If the path is a directory, use rmdir() instead.

rmdir()

Remove this directory. The directory must be empty.

lstat()

Like stat(), except if the path points to a symlink, the symlink’s status information is returned, rather than its target’s.

rename(target)

Rename this path to the given path.

replace(target)

Rename this path to the given path, clobbering the existing destination if it exists.

symlink_to(target, target_is_directory=False)

Make this path a symlink pointing to the given path. Note the order of arguments (self, target) is the reverse of os.symlink’s.

exists()

Whether this path exists.

is_dir()

Whether this path is a directory.

is_file()

Whether this path is a regular file (also True for symlinks pointing to regular files).

is_mount()

Check if this path is a POSIX mount point

is_symlink()

Whether this path is a symbolic link.

is_block_device()

Whether this path is a block device.

is_char_device()

Whether this path is a character device.

is_fifo()

Whether this path is a FIFO.

is_socket()

Whether this path is a socket.

expanduser()

Return a new path with expanded ~ and ~user constructs (as returned by os.path.expanduser)

EduNLP.ModelZoo.quesnet.quesnet.pad_packed_sequence(sequence, batch_first=False, padding_value=0.0, total_length=None)

Pads a packed batch of variable length sequences.

It is an inverse operation to pack_padded_sequence().

The returned Tensor’s data will be of size T x B x *, where T is the length of the longest sequence and B is the batch size. If batch_first is True, the data will be transposed into B x T x * format.

Example

>>> from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
>>> seq = torch.tensor([[1,2,0], [3,0,0], [4,5,6]])
>>> lens = [2, 1, 3]
>>> packed = pack_padded_sequence(seq, lens, batch_first=True, enforce_sorted=False)
>>> packed
PackedSequence(data=tensor([4, 1, 3, 5, 2, 6]), batch_sizes=tensor([3, 2, 1]),
               sorted_indices=tensor([2, 0, 1]), unsorted_indices=tensor([1, 2, 0]))
>>> seq_unpacked, lens_unpacked = pad_packed_sequence(packed, batch_first=True)
>>> seq_unpacked
tensor([[1, 2, 0],
        [3, 0, 0],
        [4, 5, 6]])
>>> lens_unpacked
tensor([2, 1, 3])

Note

total_length is useful to implement the pack sequence -> recurrent network -> unpack sequence pattern in a Module wrapped in DataParallel. See this FAQ section for details.

Parameters

• sequence (PackedSequence) – batch to pad

• batch_first (bool, optional) – if True, the output will be in B x T x * format.

• padding_value (float, optional) – values for padded elements.

• total_length (int, optional) – if not None, the output will be padded to have length total_length. This method will throw ValueError if total_length is less than the max sequence length in sequence.

Returns

Tuple of Tensor containing the padded sequence, and a Tensor containing the list of lengths of each sequence in the batch. Batch elements will be re-ordered as they were ordered originally when the batch was passed to pack_padded_sequence or pack_sequence.

class EduNLP.ModelZoo.quesnet.quesnet.PackedSequence(data, batch_sizes=None, sorted_indices=None, unsorted_indices=None)

Holds the data and list of batch_sizes of a packed sequence.

All RNN modules accept packed sequences as inputs.

Note

Instances of this class should never be created manually. They are meant to be instantiated by functions like pack_padded_sequence().

Batch sizes represent the number elements at each sequence step in the batch, not the varying sequence lengths passed to pack_padded_sequence(). For instance, given data abc and x the PackedSequence would contain data axbc with batch_sizes=[2,1,1].

data

Tensor containing packed sequence

Type

Tensor

batch_sizes

Tensor of integers holding information about the batch size at each sequence step

Type

Tensor

sorted_indices

Tensor of integers holding how this PackedSequence is constructed from sequences.

Type

Tensor, optional

unsorted_indices

Tensor of integers holding how this to recover the original sequences with correct order.

Type

Tensor, optional

Note

data can be on arbitrary device and of arbitrary dtype. sorted_indices and unsorted_indices must be torch.int64 tensors on the same device as data.

However, batch_sizes should always be a CPU torch.int64 tensor.

This invariant is maintained throughout PackedSequence class, and all functions that construct a :class:PackedSequence in PyTorch (i.e., they only pass in tensors conforming to this constraint).

pin_memory()

cuda(*args, **kwargs)

cpu(*args, **kwargs)

double()

float()

half()

long()

int()

short()

char()

byte()

to(*args, **kwargs)

Performs dtype and/or device conversion on self.data.

It has similar signature as torch.Tensor.to(), except optional arguments like non_blocking and copy should be passed as kwargs, not args, or they will not apply to the index tensors.

Note

If the self.data Tensor already has the correct torch.dtype and torch.device, then self is returned. Otherwise, returns a copy with the desired configuration.

property is_cuda

Returns true if self.data stored on a gpu

is_pinned()

Returns true if self.data stored on in pinned memory

EduNLP.ModelZoo.quesnet.quesnet.to_tensor(pic)

Convert a PIL Image or numpy.ndarray to tensor. This function does not support torchscript.

See ToTensor for more details.

Parameters

pic (PIL Image or numpy.ndarray) – Image to be converted to tensor.

Returns

Converted image.

Return type

Tensor

class EduNLP.ModelZoo.quesnet.quesnet.BiHRNN(_stoi, embs: Optional[ndarray] = None, pretrained_image: Optional[Module] = None, pretrained_meta: Optional[Module] = None, meta='know_name', emb_size=256, rnn='LSTM', lambda_input=None, lambda_loss=None, layers=4, **kwargs)

Sequence-to-sequence feature extractor based on RNN. Supports different input forms and different RNN types (LSTM/GRU),

load_emb(emb)

load_img(img_layer: Module)

load_meta(meta_layer: Module)

make_batch(data, device, pretrain=False)

Returns embeddings

forward(batch: SeqBatch)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pretrain_loss(batch)

Returns pretraining loss on a batch of data

init_h(batch_size)

classmethod from_pretrained(pretrained_dir, tokenizer)

classmethod from_config(config_path, stoi)

save(path)

Save model to path/model_name.pt and path/config.json

Parameters

path (str) – directory to save model

training: bool

save_config(config_path)

utils

class EduNLP.ModelZoo.utils.PadSequence(length, pad_val=0, clip=True)

Pad the sequence.

Pad the sequence to the given length by inserting pad_val. If clip is set, sequence that has length larger than length will be clipped.

Parameters

• length (int) – The maximum length to pad/clip the sequence

• pad_val (number) – The pad value. Default 0

• clip (bool) –

Returns

list of number

Return type

ret

EduNLP.ModelZoo.utils.pad_sequence(sequence: list, max_length=None, pad_val=0, clip=True)

Parameters

• sequence –

• max_length –

• pad_val –

• clip –

Returns

Modified list – padding the sequence in the same size.

Return type

list

Examples

>>> seq = [[4, 3, 3], [2], [3, 3, 2]]
>>> pad_sequence(seq)
[[4, 3, 3], [2, 0, 0], [3, 3, 2]]
>>> pad_sequence(seq, pad_val=1)
[[4, 3, 3], [2, 1, 1], [3, 3, 2]]
>>> pad_sequence(seq, max_length=2)
[[4, 3], [2, 0], [3, 3]]
>>> pad_sequence(seq, max_length=2, clip=False)
[[4, 3, 3], [2, 0], [3, 3, 2]]

EduNLP.ModelZoo.utils.set_device(_net, ctx, *args, **kwargs)

code from longling v1.3.26

class EduNLP.ModelZoo.utils.Masker(mask: (<class 'int'>, <class 'str'>, Ellipsis) = 0, per=0.2, seed=None)

Parameters

• mask (int, str) –

• per –

• seed –

Examples

>>> masker = Masker(per=0.5, seed=10)
>>> items = [[1, 1, 3, 4, 6], [2], [5, 9, 1, 4]]
>>> masked_seq, mask_label = masker(items)
>>> masked_seq
[[1, 1, 0, 0, 6], [2], [0, 9, 0, 4]]
>>> mask_label
[[0, 0, 1, 1, 0], [0], [1, 0, 1, 0]]
>>> items = [[1, 2, 3], [1, 1, 0], [2, 0, 0]]
>>> masked_seq, mask_label = masker(items, [3, 2, 1])
>>> masked_seq
[[1, 0, 3], [0, 1, 0], [2, 0, 0]]
>>> mask_label
[[0, 1, 0], [1, 0, 0], [0, 0, 0]]
>>> masker = Masker(mask="[MASK]", per=0.5, seed=10)
>>> items = [["a", "b", "c"], ["d", "[PAD]", "[PAD]"], ["hello", "world", "[PAD]"]]
>>> masked_seq, mask_label = masker(items, length=[3, 1, 2])
>>> masked_seq
[['a', '[MASK]', 'c'], ['d', '[PAD]', '[PAD]'], ['hello', '[MASK]', '[PAD]']]
>>> mask_label
[[0, 1, 0], [0, 0, 0], [0, 1, 0]]

Returns

list of masked_seq and list of masked_list

Return type

list

EduNLP.ModelZoo.utils.load_items(data_path)

class EduNLP.ModelZoo.utils.MLP(in_dim, n_classes, hidden_dim, dropout, n_layers=2, act=<function leaky_relu>)

forward(input)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class EduNLP.ModelZoo.utils.TextCNN(embed_dim, hidden_dim)

forward(embed)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool