Method for language-independent text tokenization using a character categorization

Patent 4991094 Issued on February 5, 1991. Estimated Expiration Date:

April 26, 2009. Estimated Expiration Date is calculated based on simple USPTO term provisions. It does not account for terminal disclaimers, term adjustments, failure to pay maintenance fees, or other factors which might affect the term of a patent.

Abstract Claims Description Full Text

Patent References

Text editing and display system
Patent #: 4057849
Issued on: 11/08/1977
Inventor: Ying , et al.

Language translating apparatus and methods therefor
Patent #: 4130882
Issued on: 12/19/1978
Inventor: Swanstrom , et al.

Associative crosspoint processor system
Patent #: 4152762
Issued on: 05/01/1979
Inventor: Bird , et al.

System for automatically hyphenating and verifying the spelling of words in a multi-lingual document
Patent #: 4456969
Issued on: 06/26/1984
Inventor: Herzik , et al.

Language translator with circuitry for detecting and holding words not stored in dictionary ROM
Patent #: 4623985
Issued on: 11/18/1986
Inventor: Morimoto , et al.

Electronic dictionary
Patent #: 4747053
Issued on: 05/24/1988
Inventor: Yoshimura , et al.

Parser for natural language text Patent #: 4887212
Issued on: 12/12/1989
Inventor: Zamora, et al.

Inventors

Application

No. 344341 filed on 04/26/1989

US Classes:

704/9, Natural language704/8Multilingual or national language support

Examiners

Primary: Fleming, Michael R.
Assistant: Sheikh, Ayaz R.

Attorney, Agent or Firm

Hoel; John E.

International Class

G06F 015/21

Abstract

A computer method is disclosed to isolate linguistically salient strings ("words") from a natural language text stream. The process is applicable to a variety of computer hardware, to any character encoding scheme, and to the idiosyncrasies of most natural languages.

Other References

M. E. Lesk et al., "Lex--A Lexical Analyzer Generator," Computer Science Technical Reports, 39, Bell Labs, Murray Hill, N.J. Oct. 1975, pp. LEX-1-LEX-1