de.tudarmstadt.ukp.dkpro.core.io.web1t.util

Class ExternalSort

java.lang.Object

de.tudarmstadt.ukp.dkpro.core.io.web1t.util.ExternalSort

public class ExternalSort
extends Object

Goal: offer a generic external-memory sorting program in Java. It must be : - hackable (easy to adapt) - scalable to large files - sensibly efficient. This software is in the public domain. Usage: java com/google/code/externalsorting/ExternalSort somefile.txt out.txt You can change the default maximal number of temporary files with the -t flag: java com/google/code/externalsorting/ExternalSort somefile.txt out.txt -t 3 For very large files, you might want to use an appropriate flag to allocate more memory to the Java VM: java -Xms2G com/google/code/externalsorting/ExternalSort somefile.txt out.txt By (in alphabetical order) Philippe Beaudoin, Jon Elsas, Christan Grant, Daniel Haran, Daniel Lemire, April 2010 originally posted at http://www.daniel-lemire.com/blog/archives/2010/04/01/external-memory-sorting-in-java/

Constructor Summary

Constructors

Constructor and Description
ExternalSort()

Method Summary

Modifier and Type	Method and Description
static long	estimateBestSizeOfBlocks(File filetobesorted, int maxtmpfiles)
static void	main(String[] args)
static int	mergeSortedFiles(List<File> files, File outputfile, Comparator<String> cmp) This merges a bunch of temporary flat files
static File	sortAndSave(List<String> tmplist, Comparator<String> cmp)
static List<File>	sortInBatch(File file, Comparator<String> cmp) This will simply load the file by blocks of x rows, then sort them in-memory, and write the result to temporary files that have to be merged later.
static List<File>	sortInBatch(File file, Comparator<String> cmp, int maxtmpfiles) This will simply load the file by blocks of x rows, then sort them in-memory, and write the result to temporary files that have to be merged later.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExternalSort

public ExternalSort()

Method Detail

estimateBestSizeOfBlocks

public static long estimateBestSizeOfBlocks(File filetobesorted,
                                            int maxtmpfiles)

sortInBatch

public static List<File> sortInBatch(File file,
                                     Comparator<String> cmp)
                              throws IOException

This will simply load the file by blocks of x rows, then sort them in-memory, and write the result to temporary files that have to be merged later.

Parameters:

file - some flat file

cmp - string comparator

Returns:

a list of temporary flat files

Throws:

IOException - if an I/O problem occurs.

sortInBatch

public static List<File> sortInBatch(File file,
                                     Comparator<String> cmp,
                                     int maxtmpfiles)
                              throws IOException

This will simply load the file by blocks of x rows, then sort them in-memory, and write the result to temporary files that have to be merged later. You can specify a bound on the number of temporary files that will be created.

Parameters:

file - some flat file

cmp - string comparator

maxtmpfiles - maximum number of temporary files

Returns:

a list of temporary flat files

Throws:

IOException - if an I/O problem occurs.

sortAndSave

public static File sortAndSave(List<String> tmplist,
                               Comparator<String> cmp)
                        throws IOException

Throws:

IOException

mergeSortedFiles

public static int mergeSortedFiles(List<File> files,
                                   File outputfile,
                                   Comparator<String> cmp)
                            throws IOException

This merges a bunch of temporary flat files

Parameters:

files - the files to merge.

outputfile - the target file.

cmp - the comprarator.

Returns:

The number of lines sorted. (P. Beaudoin)

Throws:

IOException - if an I/O problem occurs.

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException