Uses of Class opennlp.tools.tokenize.TokenSample (Apache OpenNLP Tools 2.5.9 API)
Description: use: package: opennlp.tools.tokenize, class: TokenSample

# Uses of Class
opennlp.tools.tokenize.TokenSample

Packages that use TokenSample
Package
Description
opennlp.tools.cmdline.tokenizer
opennlp.tools.formats
Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.formats.ad
Experimental package related to the `Arvores Deitadas corpus` format.
opennlp.tools.formats.conllu
Experimental package related to the CoNNL-U format.
opennlp.tools.formats.convert
opennlp.tools.formats.irishsentencebank
Experimental package related to the Irish Sentence Bank format.
opennlp.tools.formats.masc
Experimental package related to the `MASC` corpus format.
opennlp.tools.tokenize
Contains classes related to finding token or words in a string.
opennlp.tools.tokenize.lang.en

- ## Uses of TokenSample in opennlp.tools.cmdline.tokenizer

  Subclasses with type arguments of type TokenSample in opennlp.tools.cmdline.tokenizer
  Modifier and Type
  Class
  Description
  `class`
  `DetokenEvaluationErrorListener`
  A default implementation of `EvaluationMonitor` that prints to an output stream.
  `class`
  `TokenEvaluationErrorListener`
  A default implementation of `EvaluationMonitor` that prints to an output stream.
  `class`
  `TokenizerConverterTool`
  Tool to convert multiple data formats into native OpenNLP sentence detector training format.
  `final class`
  `TokenizerCrossValidatorTool`
  `final class`
  `TokenizerMEEvaluatorTool`
  A default `TokenSample`-centric implementation of `AbstractEvaluatorTool` that prints to an output stream.
  `final class`
  `TokenizerTrainerTool`
  Methods in opennlp.tools.cmdline.tokenizer with parameters of type TokenSample
  Modifier and Type
  Method
  Description
  `void`
  DetokenEvaluationErrorListener.`misclassified(TokenSample reference, TokenSample prediction)`
  `void`
  TokenEvaluationErrorListener.`misclassified(TokenSample reference, TokenSample prediction)`

- ## Uses of TokenSample in opennlp.tools.formats

  Subclasses with type arguments of type TokenSample in opennlp.tools.formats
  Modifier and Type
  Class
  Description
  `class`
  `ConllXTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  `class`
  `TokenSampleStreamFactory`
  Factory producing OpenNLP `TokenSampleStream`s.
  Methods in opennlp.tools.formats that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  ConllXTokenSampleStreamFactory.`create(String[] args)`
  `ObjectStream<TokenSample>`
  TokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.ad

  Subclasses with type arguments of type TokenSample in opennlp.tools.formats.ad
  Modifier and Type
  Class
  Description
  `class`
  `ADTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  Methods in opennlp.tools.formats.ad that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  ADTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.conllu

  Subclasses with type arguments of type TokenSample in opennlp.tools.formats.conllu
  Modifier and Type
  Class
  Description
  `class`
  `ConlluTokenSampleStream`
  `class`
  `ConlluTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  Methods in opennlp.tools.formats.conllu that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  ConlluTokenSampleStream.`read()`
  Methods in opennlp.tools.formats.conllu that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  ConlluTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.convert

  Subclasses with type arguments of type TokenSample in opennlp.tools.formats.convert
  Modifier and Type
  Class
  Description
  `class`
  `NameToTokenSampleStream`
  Note: Do not use this class, internal use only!
  `class`
  `NameToTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  `class`
  `ParseToTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  `class`
  `POSToTokenSampleStream`
  Note: Do not use this class, internal use only!
  `class`
  `POSToTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  Methods in opennlp.tools.formats.convert that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  NameToTokenSampleStream.`read()`
  `TokenSample`
  POSToTokenSampleStream.`read()`
  Methods in opennlp.tools.formats.convert that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  NameToTokenSampleStreamFactory.`create(String[] args)`
  `ObjectStream<TokenSample>`
  ParseToTokenSampleStreamFactory.`create(String[] args)`
  `ObjectStream<TokenSample>`
  POSToTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.irishsentencebank

  Subclasses with type arguments of type TokenSample in opennlp.tools.formats.irishsentencebank
  Modifier and Type
  Class
  Description
  `class`
  `IrishSentenceBankTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  Methods in opennlp.tools.formats.irishsentencebank that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  IrishSentenceBankDocument.IrishSentenceBankSentence.`getTokenSample()`
  Methods in opennlp.tools.formats.irishsentencebank that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  IrishSentenceBankTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.masc

  Subclasses with type arguments of type TokenSample in opennlp.tools.formats.masc
  Modifier and Type
  Class
  Description
  `class`
  `MascTokenSampleStream`
  `class`
  `MascTokenSampleStreamFactory`
  Note: Do not use this class, internal use only!
  Methods in opennlp.tools.formats.masc that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  MascTokenSampleStream.`read()`
  Methods in opennlp.tools.formats.masc that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  MascTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.tokenize

  Subclasses with type arguments of type TokenSample in opennlp.tools.tokenize
  Modifier and Type
  Class
  Description
  `class`
  `DetokenizerEvaluator`
  The `DetokenizerEvaluator` measures the performance of the given `Detokenizer` with the provided reference `samples`.
  `class`
  `TokenizerEvaluator`
  The `TokenizerEvaluator` measures the performance of the given `Tokenizer` with the provided reference `samples`.
  `class`
  `TokenSampleStream`
  This class is a `stream filter` which reads in string encoded samples and creates `samples` out of them.
  `class`
  `TokSpanEventStream`
  This class reads the `samples` via an `Iterator` and converts the samples into `events` which can be used by the maxent library for training.
  `class`
  `WhitespaceTokenStream`
  This stream formats `ObjectStream` of `samples` into whitespace separated token strings.
  Subinterfaces with type arguments of type TokenSample in opennlp.tools.tokenize
  Modifier and Type
  Interface
  Description
  `interface`
  `TokenizerEvaluationMonitor`
  A marker interface for evaluating `tokenizers`.
  Classes in opennlp.tools.tokenize that implement interfaces with type arguments of type TokenSample
  Modifier and Type
  Class
  Description
  `class`
  `TokenizerStream`
  The `TokenizerStream` uses a `Tokenizer` to tokenize the input string and output `samples`.
  Methods in opennlp.tools.tokenize that return TokenSample
  Modifier and Type
  Method
  Description
  `static TokenSample`
  TokenSample.`parse(String sampleString, String separatorChars)`
  Parses a string sample.
  `protected TokenSample`
  DetokenizerEvaluator.`processSample(TokenSample reference)`
  `protected TokenSample`
  TokenizerEvaluator.`processSample(TokenSample reference)`
  `TokenSample`
  TokenizerStream.`read()`
  `TokenSample`
  TokenSampleStream.`read()`
  Methods in opennlp.tools.tokenize with parameters of type TokenSample
  Modifier and Type
  Method
  Description
  `protected Iterator<Event>`
  TokSpanEventStream.`createEvents(TokenSample tokenSample)`
  Adds training events to the event stream for each of the specified `sample`.
  `protected TokenSample`
  DetokenizerEvaluator.`processSample(TokenSample reference)`
  `protected TokenSample`
  TokenizerEvaluator.`processSample(TokenSample reference)`
  Method parameters in opennlp.tools.tokenize with type arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `void`
  TokenizerCrossValidator.`evaluate(ObjectStream<TokenSample> samples, int nFolds)`
  Starts the evaluation.
  `static TokenizerModel`
  TokenizerME.`train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams)`
  Trains a model for the `TokenizerME`.
  Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSample
  Modifier
  Constructor
  Description
  ``
  `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics)`
  Initializes a new event stream based on the data stream using a `TokenContextGenerator` that relies on a `DefaultTokenContextGenerator`.
  ``
  `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg)`
  Initializes a new event stream based on the data stream using a `TokenContextGenerator`.
  ``
  `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg)`
  Initializes a new event stream based on the data stream using a `TokenContextGenerator`.
  ``
  `WhitespaceTokenStream(ObjectStream<TokenSample> tokens)`
  Initializes a `WhitespaceTokenStream`.

- ## Uses of TokenSample in opennlp.tools.tokenize.lang.en

  Classes in opennlp.tools.tokenize.lang.en that implement interfaces with type arguments of type TokenSample
  Modifier and Type
  Class
  Description
  `class`
  `TokenSampleStream`
  Class which produces an Iterator<TokenSample> from a file of space delimited token.
  Methods in opennlp.tools.tokenize.lang.en that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  TokenSampleStream.`next()`