remove stop words from documents -凯发k8网页登录

remove the stop words from an array of documents using removestopwords. the tokenizeddocument function detects that the documents are in english, so removestopwords removes english stop words.

documents = tokenizeddocument([
    "an example of a short sentence" 
    "a second short sentence"]);
newdocuments = removestopwords(documents)

newdocuments = 
  2x1 tokenizeddocument:
    3 tokens: example short sentence
    3 tokens: second short sentence

tokenize japanese text using tokenizeddocument. the function automatically detects japanese text.

str = [
    "ここは静かなので、とても穏やかです"
    "企業内の顧客データを利用し、今年の売り上げを調べることが出来た。"
    "私は先生です。私は英語を教えています。"];
documents = tokenizeddocument(str);

remove stop words using removestopwords. the function uses the language details from documents to determine which language stop words to remove.

documents = removestopwords(documents)

documents = 
  3x1 tokenizeddocument:
     4 tokens: 静か 、 とても 穏やか
    10 tokens: 企業 顧客 データ 利用 、 今年 売り上げ 調べる 出来 。
     5 tokens: 先生 。 英語 教え 。

tokenize german text using tokenizeddocument. the function automatically detects german text.

str = [
    "guten morgen. wie geht es dir?"
    "heute wird ein guter tag."];
documents = tokenizeddocument(str)

documents = 
  2x1 tokenizeddocument:
    8 tokens: guten morgen . wie geht es dir ?
    6 tokens: heute wird ein guter tag .

remove stop words using the removestopwords function. the function uses the language details from documents to determine which language stop words to remove.

documents = removestopwords(documents)

documents = 
  2x1 tokenizeddocument:
    5 tokens: guten morgen . geht ?
    5 tokens: heute wird guter tag .