EduNLP.I2V¶

class EduNLP.I2V.i2v.I2V(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

It just a api, so you shouldn’t use it directly. If you want to get vector from item, you can use other model like D2V and W2V.

Parameters

tokenizer (str) – the tokenizer name
t2v (str) – the name of token2vector model
args – the parameters passed to t2v
tokenizer_kwargs (dict) – the parameters passed to tokenizer
pretrained_t2v (bool) –
- True: use pretrained t2v model
- False: use your own t2v model
kwargs – the parameters passed to t2v

Examples

>>> item = {"如图来自古希腊数学家希波克拉底所研究的几何图形．此图由三个半圆构成，三个半圆的直径分别为直角三角形$ABC$的斜边$BC$,     ... 直角边$AB$, $AC$.$\bigtriangleup ABC$的三边所围成的区域记为$I$,黑色部分记为$II$, 其余部分记为$III$.在整个图形中随机取一点，    ... 此点取自$I,II,III$的概率分别记为$p_1,p_2,p_3$,则$\SIFChoice$$\FigureID{1}$"}
>>> model_path = "examples/test_model/d2v/test_gensim_luna_stem_tf_d2v_256.bin"
>>> i2v = D2V("text","d2v",filepath=model_path, pretrained_t2v = False)
>>> i2v(item)
([array([ ...dtype=float32)], None)

Returns: i2v model
Return type: I2V

tokenize(items, *args, indexing=True, padding=False, key=<function I2V.<lambda>>, **kwargs) → list[source]¶

infer_vector(items, tokenize=True, indexing=False, padding=False, key=<function I2V.<lambda>>, **kwargs) → tuple[source]¶

infer_item_vector(tokens, *args, **kwargs) → ...[source]¶

infer_token_vector(tokens, *args, **kwargs) → ...[source]¶

save(config_path)[source]¶

classmethod load(config_path, *args, **kwargs)[source]¶

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', *args, **kwargs)[source]¶

property vector_size¶

class EduNLP.I2V.i2v.D2V(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

The model aims to transfer item to vector directly.

I2V

Parameters

tokenizer (str) – the tokenizer name
t2v (str) – the name of token2vector model
args – the parameters passed to t2v
tokenizer_kwargs (dict) – the parameters passed to tokenizer
pretrained_t2v (bool) – True: use pretrained t2v model False: use your own t2v model
kwargs – the parameters passed to t2v

Examples

>>> item = {"如图来自古希腊数学家希波克拉底所研究的几何图形．此图由三个半圆构成，三个半圆的直径分别为直角三角形$ABC$的斜边$BC$,     ... 直角边$AB$, $AC$.$\bigtriangleup ABC$的三边所围成的区域记为$I$,黑色部分记为$II$, 其余部分记为$III$.在整个图形中随机取一点，    ... 此点取自$I,II,III$的概率分别记为$p_1,p_2,p_3$,则$\SIFChoice$$\FigureID{1}$"}
>>> model_path = "examples/test_model/d2v/d2v_test_256/d2v_test_256.bin"
>>> i2v = D2V("text","d2v",filepath=model_path, pretrained_t2v = False)
>>> i2v(item)
([array([ ...dtype=float32)], None)

Returns: i2v model
Return type: I2V

infer_vector(items, tokenize=True, indexing=False, padding=False, key=<function D2V.<lambda>>, *args, **kwargs) → tuple[source]¶

It is a function to switch item to vector. And before using the function, it is necessary to load model.

Parameters

items (str) – the text of question
tokenize (bool) – True: tokenize the item
indexing (bool) –
padding (bool) –
key (lambda function) – the parameter passed to tokenizer, select the text to be processed
args – the parameters passed to t2v
kwargs – the parameters passed to t2v

Returns

vector

Return type

list

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', *args, **kwargs)[source]¶

class EduNLP.I2V.i2v.W2V(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

The model aims to transfer tokens to vector.

I2V

Parameters

tokenizer (str) – the tokenizer name
t2v (str) – the name of token2vector model
args – the parameters passed to t2v
tokenizer_kwargs (dict) – the parameters passed to tokenizer
pretrained_t2v (bool) – True: use pretrained t2v model False: use your own t2v model
kwargs – the parameters passed to t2v

Examples

>>> (); i2v = get_pretrained_i2v("w2v_test_256", "examples/test_model/w2v"); () 
(...)
>>> item_vector, token_vector = i2v(["有学者认为：‘学习’，必须适应实际"])
>>> item_vector 
[array([...], dtype=float32)]

Returns: i2v model
Return type: W2V

infer_vector(items, tokenize=True, indexing=False, padding=False, key=<function W2V.<lambda>>, *args, **kwargs) → tuple[source]¶

It is a function to switch item to vector. And before using the function, it is necessary to load model.

Parameters

items (str) – the text of question
tokenize (bool) – True: tokenize the item
indexing (bool) –
padding (bool) –
key (lambda function) – the parameter passed to tokenizer, select the text to be processed
args – the parameters passed to t2v
kwargs – the parameters passed to t2v

Returns

vector

Return type

list

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', *args, **kwargs)[source]¶

class EduNLP.I2V.i2v.Elmo(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

The model aims to transfer item and tokens to vector with Elmo.

I2V

Parameters

tokenizer (str) – the tokenizer name
t2v (str) – the name of token2vector model
args – the parameters passed to t2v
tokenizer_kwargs (dict) – the parameters passed to tokenizer
pretrained_t2v (bool) – True: use pretrained t2v model False: use your own t2v model
kwargs – the parameters passed to t2v

Returns

i2v model

Return type

Elmo

infer_vector(items, tokenize=True, return_tensors='pt', *args, **kwargs) → tuple[source]¶

It is a function to switch item to vector. And before using the function, it is necessary to load model.

Parameters

items (str or list) – the text of question
tokenize (bool) – True: tokenize the item
return_tensors (str) – tensor type used in tokenizer
args – the parameters passed to t2v
kwargs – the parameters passed to t2v

Returns

vector

Return type

list

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', *args, **kwargs)[source]¶

class EduNLP.I2V.i2v.Bert(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

The model aims to transfer item and tokens to vector with Bert.

I2V

Parameters

tokenizer (str) – the tokenizer name
t2v (str) – the name of token2vector model
args – the parameters passed to t2v
tokenizer_kwargs (dict) – the parameters passed to tokenizer
pretrained_t2v (bool) – True: use pretrained t2v model False: use your own t2v model
kwargs – the parameters passed to t2v

Returns

i2v model

Return type

Bert

infer_vector(items, tokenize=True, return_tensors='pt', *args, **kwargs) → tuple[source]¶

It is a function to switch item to vector. And before using the function, it is nesseary to load model.

Parameters

items (str or list) – the text of question
tokenize (bool) – True: tokenize the item
return_tensors (str) – tensor type used in tokenizer
args – the parameters passed to t2v
kwargs – the parameters passed to t2v

Returns

vector

Return type

list

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', *args, **kwargs)[source]¶

class EduNLP.I2V.i2v.DisenQ(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

The model aims to transfer item and tokens to vector with DisenQ. Bases ——- I2V :param tokenizer: the tokenizer name :type tokenizer: str :param t2v: the name of token2vector model :type t2v: str :param args: the parameters passed to t2v :param tokenizer_kwargs: the parameters passed to tokenizer :type tokenizer_kwargs: dict :param pretrained_t2v: True: use pretrained t2v model

False: use your own t2v model

Parameters: kwargs – the parameters passed to t2v
Returns: i2v model
Return type: DisenQ

infer_vector(items: (<class 'dict'>, <class 'list'>), tokenize=True, key=<function DisenQ.<lambda>>, vector_type=None, **kwargs) → tuple[source]¶

It is a function to switch item to vector. And before using the function, it is nesseary to load model. :param item: the item of question :type item: dict or list :param tokenize: True: tokenize the item :type tokenize: bool :param key: the parameter passed to tokenizer, select the text to be processed :type key: lambda function :param args: the parameters passed to t2v :param kwargs: the parameters passed to t2v

Returns: vector
Return type: list

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', **kwargs)[source]¶

class EduNLP.I2V.i2v.QuesNet(tokenizer, t2v, *args, tokenizer_kwargs: Optional[dict] = None, pretrained_t2v=False, **kwargs)[source]¶

The model aims to transfer item and tokens to vector with quesnet. Bases ——- I2V

infer_vector(item, tokenize=True, key=<function QuesNet.<lambda>>, meta=['know_name'], *args, **kwargs)[source]¶

It is a function to switch item to vector. And before using the function, it is nesseary to load model. :param item: the item of question, or question list :type item: str or dict or list :param tokenize: True: tokenize the item :type tokenize: bool, optional :param key: _description_, by default lambdax:x :type key: _type_, optional :param meta: meta information, by default [‘know_name’] :type meta: list, optional :param args: the parameters passed to t2v :param kwargs: the parameters passed to t2v

Returns

token embeddings
question embedding

classmethod from_pretrained(name, model_dir='/home/docs/.EduNLP/model', *args, **kwargs)[source]¶

EduNLP.I2V.i2v.get_pretrained_i2v(name, model_dir='/home/docs/.EduNLP/model')[source]¶

It is a good idea if you want to switch item to vector earily.

Parameters

name (str) – the name of item2vector model e.g.: d2v_math_300 w2v_math_300 elmo_math_2048 bert_math_768 bert_taledu_768 disenq_math_256 quesnet_math_512
model_dir (str) – the path of model, default: MODEL_DIR = ‘~/.EduNLP/model’

Returns

i2v model

Return type

I2V

Examples

>>> item = {"如图来自古希腊数学家希波克拉底所研究的几何图形．此图由三个半圆构成，三个半圆的直径分别为直角三角形$ABC$的斜边$BC$,     ... 直角边$AB$, $AC$.$\bigtriangleup ABC$的三边所围成的区域记为$I$,黑色部分记为$II$, 其余部分记为$III$.在整个图形中随机取一点，    ... 此点取自$I,II,III$的概率分别记为$p_1,p_2,p_3$,则$\SIFChoice$$\FigureID{1}$"}
>>> (); i2v = get_pretrained_i2v("d2v_test_256", "examples/test_model/d2v"); () 
(...)
>>> print(i2v(item))
([array([ ...dtype=float32)], None)