org.dkpro.core.api.embeddings.binary

Class BinaryWordVectorUtils

java.lang.Object

org.dkpro.core.api.embeddings.binary.BinaryWordVectorUtils

public class BinaryWordVectorUtils
extends Object

Utility Methods for working with binary dl4j word vector files.

The core of this code has been written in the context of dl4j, but provides a generic solution to efficiently storing and reading word embeddings with a memory-mapped file.

Author:

Paul Dubs, Richard Eckart de Castilho

See Also:

Original source

Constructor Summary

Constructors

Constructor and Description
BinaryWordVectorUtils()

Method Summary

Modifier and Type	Method and Description
static void	convertWordVectorsToBinary(Map<String,float[]> vectors, boolean aCaseless, Locale aLocale, File binaryTarget) Write a map of token embeddings into binary format.
static void	convertWordVectorsToBinary(Map<String,float[]> vectors, File binaryTarget) Write a map of token embeddings into binary format.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BinaryWordVectorUtils

public BinaryWordVectorUtils()

Method Detail

convertWordVectorsToBinary

public static void convertWordVectorsToBinary(Map<String,float[]> vectors,
                                              File binaryTarget)
                                       throws IOException

Write a map of token embeddings into binary format. Uses the default locale Locale.US and assume case-sensitivity iff there is any token containing an uppercase letter.

Parameters:

vectors - a Map<String, float[]> holding all tokens with embeddings

binaryTarget - the target file File

Throws:

IOException - if an I/O error occurs

See Also:

convertWordVectorsToBinary(Map, boolean, Locale, File)

convertWordVectorsToBinary

public static void convertWordVectorsToBinary(Map<String,float[]> vectors,
                                              boolean aCaseless,
                                              Locale aLocale,
                                              File binaryTarget)
                                       throws IOException

Write a map of token embeddings into binary format.

Parameters:

vectors - a Map<String, float[]> holding all tokens with embeddings

aCaseless - if true, tokens are expected to be caseless

aLocale - the Locale

binaryTarget - the target file File

Throws:

IOException - if an I/O error occurs