org.dkpro.core.api.embeddings.text

Class TextFormatVectorizerUtils

java.lang.Object

org.dkpro.core.api.embeddings.text.TextFormatVectorizerUtils

public class TextFormatVectorizerUtils
extends Object

Helper Methods for reading word embeddings from text format file and converting into binary format.

Constructor Summary

Constructors

Constructor and Description
TextFormatVectorizerUtils()

Method Summary

Modifier and Type	Method and Description
static void	convertMalletEmbeddingsToBinary(File malletEmbeddings, boolean aCaseless, Locale aLocale, File targetFile) Read a (compressed) Mallet embeddings file (in text format) and convert it into the binary format using BinaryWordVectorUtils.
static void	convertMalletEmbeddingsToBinary(File malletEmbeddings, File targetFile) Read a (compressed) Mallet embeddings file (in text format) and convert it into the binary format using BinaryWordVectorUtils.
static Map<String,float[]>	readEmbeddingFileTxt(File file, boolean hasHeader) Read an embeddings file in text format.
static Map<String,float[]>	readEmbeddingFileTxt(InputStream inputStream, boolean hasHeader) Read embeddings in text format from an InputStream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextFormatVectorizerUtils

public TextFormatVectorizerUtils()

Method Detail

readEmbeddingFileTxt

public static Map<String,float[]> readEmbeddingFileTxt(File file,
                                                       boolean hasHeader)
                                                throws IOException

Read an embeddings file in text format.

If hasHeader is set to true, the first line is expected to contain the size and dimensionality of the vectors. This is typically true for files generated by Word2Vec (in text format).

Parameters:

file - the input file

hasHeader - if true, read size and dimensionality from the first line

Returns:

a Map<String, float[]> mapping each token to a vector.

Throws:

IOException - if the input file cannot be read

See Also:

readEmbeddingFileTxt(InputStream, boolean)

readEmbeddingFileTxt

public static Map<String,float[]> readEmbeddingFileTxt(InputStream inputStream,
                                                       boolean hasHeader)
                                                throws IOException

Read embeddings in text format from an InputStream. Each line is expected to have a whitespace-separated list <token> <value1> <value2> ....

Parameters:

inputStream - an InputStream

hasHeader - if true, read size and dimensionality from the first line

Returns:

a Map<String, float[]> mapping each token to a vector.

Throws:

IOException - if the input file cannot be read

convertMalletEmbeddingsToBinary

public static void convertMalletEmbeddingsToBinary(File malletEmbeddings,
                                                   File targetFile)
                                            throws IOException

Read a (compressed) Mallet embeddings file (in text format) and convert it into the binary format using BinaryWordVectorUtils.

Parameters:

malletEmbeddings - a File holding embeddings in text format

targetFile - the output File

Throws:

IOException - if an I/O error occurs.

convertMalletEmbeddingsToBinary

public static void convertMalletEmbeddingsToBinary(File malletEmbeddings,
                                                   boolean aCaseless,
                                                   Locale aLocale,
                                                   File targetFile)
                                            throws IOException

Read a (compressed) Mallet embeddings file (in text format) and convert it into the binary format using BinaryWordVectorUtils.

Parameters:

malletEmbeddings - a File holding embeddings in text format

aCaseless - if true, all input tokens are expected to be caseless

aLocale - the Locale to use

targetFile - the output File

Throws:

IOException - if an I/O error occurs.