document similarities with bm25 algorithm -凯发k8网页登录

create an array of tokenized documents.

textdata = [
    "the quick brown fox jumped over the lazy dog"
    "the fast brown fox jumped over the lazy dog"
    "the lazy dog sat there and did nothing"
    "the other animals sat there watching"];
documents = tokenizeddocument(textdata)

documents = 
  4x1 tokenizeddocument:
    9 tokens: the quick brown fox jumped over the lazy dog
    9 tokens: the fast brown fox jumped over the lazy dog
    8 tokens: the lazy dog sat there and did nothing
    6 tokens: the other animals sat there watching

calculate the similarities between them using the bm25similarity function. the output is a sparse matrix.

similarities = bm25similarity(documents);

visualize the similarities of the documents in a heat map.

figure
heatmap(similarities);
xlabel("document")
ylabel("document")
title("bm25 similarities")

the first three documents have the highest pairwise similarities which indicates that these documents are most similar. the last document has comparatively low pairwise similarities with the other documents which indicates that this document is less like the other documents.

create an array of input documents.

str = [
    "the quick brown fox jumped over the lazy dog"
    "the fast fox jumped over the lazy dog"
    "the dog sat there and did nothing"
    "the other animals sat there watching"];
documents = tokenizeddocument(str)

documents = 
  4x1 tokenizeddocument:
    9 tokens: the quick brown fox jumped over the lazy dog
    8 tokens: the fast fox jumped over the lazy dog
    7 tokens: the dog sat there and did nothing
    6 tokens: the other animals sat there watching

create an array of query documents.

str = [
    "a brown fox leaped over the lazy dog"
    "another fox leaped over the dog"];
queries = tokenizeddocument(str)

queries = 
  2x1 tokenizeddocument:
    8 tokens: a brown fox leaped over the lazy dog
    6 tokens: another fox leaped over the dog

calculate the similarities between input documents and query documents using the bm25similarity function. the output is a sparse matrix. the score in similarities(i,j) represents the similarity between documents(i) and queries(j).

similarities = bm25similarity(documents,queries);

visualize the similarities of the documents in a heat map.

figure
heatmap(similarities);
xlabel("query document")
ylabel("input document")
title("bm25 similarities")

in this case, the first input document is most like the first query document.

create a bag-of-words model from the text data in sonnets.csv.

filename = "sonnets.csv";
tbl = readtable(filename,'texttype','string');
textdata = tbl.sonnet;
documents = tokenizeddocument(textdata);
bag = bagofwords(documents)

bag = 
  bagofwords with properties:
          counts: [154x3527 double]
      vocabulary: ["from"    "fairest"    "creatures"    "we"    "desire"    "increase"    ","    "that"    "thereby"    "beauty's"    "rose"    "might"    "never"    "die"    "but"    "as"    "the"    "riper"    "should"    "by"    "time"    ...    ]
        numwords: 3527
    numdocuments: 154

calculate similarities between the sonnets using the bm25similarity function. the output is a sparse matrix.

similarities = bm25similarity(bag);

visualize the similarities between the first five documents in a heat map.

figure
heatmap(similarities(1:5,1:5));
xlabel("document")
ylabel("document")
title("bm25 similarities")

the bm25 algorithm addresses a limitation of the bm25 algorithm: the component of the term-frequency normalization by document length is not properly lower bounded. as a result of this limitation, long documents which do not match the query term can often be scored unfairly by bm25 as having a similar relevance to shorter documents that do not contain the query term.

bm25 addresses this limitation by using a document length correction factor (the value of the 'documentlengthscaling' name-value pair). this factor prevents the algorithm from over-penalizing long documents.

create two arrays of tokenized documents.

textdata1 = [
    "the quick brown fox jumped over the lazy dog"
    "the fast fox jumped over the lazy dog"
    "the dog sat there and did nothing"
    "the other animals sat there watching"];
documents1 = tokenizeddocument(textdata1)

documents1 = 
  4x1 tokenizeddocument:
    9 tokens: the quick brown fox jumped over the lazy dog
    8 tokens: the fast fox jumped over the lazy dog
    7 tokens: the dog sat there and did nothing
    6 tokens: the other animals sat there watching

textdata2 = [
    "a brown fox leaped over the lazy dog"
    "another fox leaped over the dog"];
documents2 = tokenizeddocument(textdata2)

documents2 = 
  2x1 tokenizeddocument:
    8 tokens: a brown fox leaped over the lazy dog
    6 tokens: another fox leaped over the dog

to calculate the bm25 document similarities, use the bm25similarity function and set the 'documentlengthcorrection' option to a nonzero value. in this case, set the 'documentlengthcorrection' option to 1.

similarities = bm25similarity(documents1,documents2,'documentlengthcorrection',1);

visualize the similarities of the documents in a heat map.

figure
heatmap(similarities);
xlabel("query")
ylabel("document")
title("bm25  similarities")

here, when compared with the example similarity between documents, the scores show more similarity between the input documents and the first query document.