'payload'에 해당되는 글 3건

  1. 2016.02.23 [Elasticsearch] Delimited Payload Token Filter
  2. 2013.07.15 [lucene] [펌]PayloadTermQuery
  3. 2013.07.15 [Elasticsearch] [링크 펌]Custom Similarity...

[Elasticsearch] Delimited Payload Token Filter

Elastic/Elasticsearch 2016. 2. 23. 16:31

2.X 올라가면서 전체 API를 살펴 보지 못했는데 형분기 관련 구성을 하다가 눈에 확 들어 오는게 있어서 기록해 봅니다.


원문링크)

https://www.elastic.co/guide/en/elasticsearch/reference/2.2/analysis-delimited-payload-tokenfilter.html


원문스크랩)

Named delimited_payload_filter. Splits tokens into tokens and payload whenever a delimiter character is found.


Example: "the|1 quick|2 fox|3" is split by default into tokens the, quick, and fox with payloads 1, 2, and 3 respectively.


Parameters:


delimiter

  Character used for splitting the tokens. Default is |.

encoding

  The type of the payload. int for integer, float for float and identity for characters. Default is float.



예전에 문서가 가지는 별도의 rankin, boostin, keyword score 등등 검색 로그를 기반으로 문서 가중치, 랭킹, 추천 데이터를 생성해서 문서에 반영해서 질의 시점에 사용을 했었는데요.

이 token filter 가 초기 0.90 사용할때 없어서 별도로 script plugin을 만들어서 사용했었습니다.

그렇다 보니 string 연산을 script 내부에서 하니까 성능적으로 문제가 있었는데요. 이 기능을 사용하면 성능 이슈 없이 쉽게 구현할 수 있을 것 같습니다.


1.3 부터 들어온 API 인것 같은데 그 동안 왜 몰랐나 싶내요.

한번 실험해 보고 결과도 공유 하도록 하겠습니다.

저작자표시 비영리 변경금지
:

[lucene] [펌]PayloadTermQuery

Elastic/Elasticsearch 2013. 7. 15. 23:29

http://sujitpal.blogspot.kr/2010/10/denormalizing-maps-with-lucene-payloads.html


Friday, October 08, 2010

Denormalizing Maps with Lucene Payloads

Last week, I tried out Lucene's Payload and SpanQuery features to do some position based custom scoring of terms. I've been interested in the Payload feature ever since I first read about it, because it looked like something I could use to solve another problem at work...

The problem is to to be able to store a mapping of concepts to scores along with a document. Our search uses a medical taxonomy, basically a graph of medical concepts (nodes) and their relationships to each other (edges). During indexing, a document is analyzed and a map of node IDs and scores is created and stored in the index. The score is composed of various components, but for simplicity, it can be thought of as the number of occurrences of a node in the document. So after indexing, we would end up with something like this:

During search, the query is decomposed into concepts using a similar process, and a query consisting of one or more TermQueries (wrapped in a BooleanQuery) are used to pull documents out of the index. In pseudo-SQL, something like this:

1
2
3
4
5
  SELECT document FROM index
  WHERE nodeId = nodeID(1)
  ...
  AND/OR nodeId = nodeID(n)
  ORDER by (score(1) + score(n)) DESC

There are many approaches to efficiently model this sort of situation, and over the years we've tried a few. The approach I am going to describe uses Lucene's Payload feature. Basically, the concept map is "flattened" into the main Document, and the scores are farmed out to a Payload byte array, so we can use the scores for scoring our results.

Obviously, this is nothing new... other people have used Payloads to do very similar things. In fact, a lot of the code that follows is heavily based on the example in this Lucid Imagination blog post.

Indexing

At index time, we flatten our concept map into a whitespace separated list of key-value pairs, and the key and value in each element is separated out with a special character, in our case a "$" sign. So a concept map {p1 => 123.0, p2 => 234.0} would be transformed to "p1$123.0 p2$234.0".

Lucene provides the DelimitedPayloadTokenFilter, a custom TokenFilter to parse this string and convert it to equivalent term and payload pairs, so all we have to build on our own is our custom Analyzer. The IndexWriter will use this custom Analyzer for the "data" field in the JUnit test (see below).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
// Source: src/main/java/com/mycompany/payload/MyPayloadAnalyzer.java
package com.mycompany.payload;

import java.io.Reader;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.WhitespaceTokenizer;
import org.apache.lucene.analysis.payloads.DelimitedPayloadTokenFilter;
import org.apache.lucene.analysis.payloads.FloatEncoder;

public class MyPayloadAnalyzer extends Analyzer {

  @Override
  public TokenStream tokenStream(String fieldName, Reader reader) {
    return new DelimitedPayloadTokenFilter(
      new WhitespaceTokenizer(reader),
      '$', new FloatEncoder());
  }
}

Searching

On the search side, we create a custom Similarity implementation that reads the score from the payload and returns it. We will tell our searcher to use this Similarity implementation. We want to use only ourconcept scores, not make it part of the full Lucene score, so we indicate that when we create our PayloadTermQuery in our JUnit test.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
// Source: src/main/java/com/mycompany/payload/MyPayloadSimilarity.java
package com.mycompany.payload;

import org.apache.lucene.analysis.payloads.PayloadHelper;
import org.apache.lucene.search.DefaultSimilarity;

public class MyPayloadSimilarity extends DefaultSimilarity {

  private static final long serialVersionUID = -2402909220013794848L;

  @Override
  public float scorePayload(int docId, String fieldName,
      int start, int end, byte[] payload, int offset, int length) {
    if (payload != null) {
      return PayloadHelper.decodeFloat(payload, offset);
    } else {
      return 1.0F;
    }
  }
}

The actual search logic is in the JUnit test shown below. Here I build a small index with some dummy data in RAM and query it using a straight PayloadTermQuery and two Boolean queries with embedded PayloadTermQueries.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
// Source: src/test/java/com/mycompany/payload/MyPayloadQueryTest.java
package com.mycompany.payload;

import org.apache.commons.lang.StringUtils;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.Field.Index;
import org.apache.lucene.document.Field.Store;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.payloads.AveragePayloadFunction;
import org.apache.lucene.search.payloads.PayloadTermQuery;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;

public class MyPayloadQueryTest {

  private static IndexSearcher searcher;
  
  private static String[] data = {
    "p1$123.0 p2$2.0 p3$89.0",
    "p2$91.0 p1$5.0",
    "p3$56.0 p1$25.0",
    "p4$98.0 p5$65.0 p1$33.0"
  };

  @BeforeClass
  public static void setupBeforeClass() throws Exception {
    Directory directory = new RAMDirectory();
    IndexWriter writer = new IndexWriter(directory, 
      new MyPayloadAnalyzer(), IndexWriter.MaxFieldLength.UNLIMITED);;
    for (int i = 0; i < data.length; i++) {
      Document doc = new Document();
      doc.add(new Field("title", "Document #" + i, Store.YES, Index.NO));
      doc.add(new Field("data", data[i], Store.YES, Index.ANALYZED));
      writer.addDocument(doc);
    }
    writer.close();
    searcher = new IndexSearcher(directory);
    searcher.setSimilarity(new MyPayloadSimilarity());
  }

  @AfterClass
  public static void teardownAfterClass() throws Exception {
    if (searcher != null) {
      searcher.close();
    }
  }
  
  @Test
  public void testSingleTerm() throws Exception {
    PayloadTermQuery p1Query = new PayloadTermQuery(
      new Term("data", "p1"), new AveragePayloadFunction(), false);
    search(p1Query);
  }
  
  @Test
  public void testAndQuery() throws Exception {
    PayloadTermQuery p1Query = new PayloadTermQuery(
      new Term("data", "p1"), new AveragePayloadFunction(), false);
    PayloadTermQuery p2Query = new PayloadTermQuery(
      new Term("data", "p2"), new AveragePayloadFunction(), false);
    BooleanQuery query = new BooleanQuery();
    query.add(p1Query, Occur.MUST);
    query.add(p2Query, Occur.MUST);
    search(query);
  }
  
  @Test
  public void testOrQuery() throws Exception {
    PayloadTermQuery p1Query = new PayloadTermQuery(
      new Term("data", "p1"), new AveragePayloadFunction(), false);
    PayloadTermQuery p2Query = new PayloadTermQuery(
      new Term("data", "p2"), new AveragePayloadFunction(), false);
    BooleanQuery query = new BooleanQuery();
    query.add(p1Query, Occur.SHOULD);
    query.add(p2Query, Occur.SHOULD);
    search(query);
  }
  
  private void search(Query query) throws Exception {
    System.out.println("=== Running query: " + query.toString() + " ===");
    ScoreDoc[] hits = searcher.search(query, 10).scoreDocs;
    for (int i = 0; i < hits.length; i++) {
      Document doc = searcher.doc(hits[i].doc);
      System.out.println(StringUtils.join(new String[] {
        doc.get("title"),
        doc.get("data"),
        String.valueOf(hits[i].score)
      }, "  "));
    }
  }
}

The three tests (annotated with @Test) cover the basic use cases that I expect for this search - a single term search, an AND term search and an OR term search. The last two are done by embedding the individual PayloadTermQuery objects into a BooleanQuery. As you can see from the results below, this works quite nicely. This is good news for me, since based on my reading of the LIA2 book, I had (wrongly) concluded that Payloads can only be used with SpanQuery, and that you need special "payload aware" subclasses of SpanQuery to be able to use them (which is true in case of SpanQuery, BTW).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
=== Running query: data:p1 ===
Document #0  p1$123.0 p2$2.0 p3$89.0  123.0
Document #3  p4$98.0 p5$65.0 p1$33.0  33.0
Document #2  p3$56.0 p1$25.0  25.0
Document #1  p2$91.0 p1$5.0  5.0

=== Running query: +data:p1 +data:p2 ===
Document #0  p1$123.0 p2$2.0 p3$89.0  125.0
Document #1  p2$91.0 p1$5.0  96.0

=== Running query: data:p1 data:p2 ===
Document #0  p1$123.0 p2$2.0 p3$89.0  125.0
Document #1  p2$91.0 p1$5.0  96.0
Document #3  p4$98.0 p5$65.0 p1$33.0  16.5
Document #2  p3$56.0 p1$25.0  12.5

Performance

I also read (on the web, can't find the link now) that Payload queries are usually slower than their non-payload aware counterparts, so I decided to do a quick back-of-the-envelope calculation to see what sort of degradation to expect.

I took an existing index containing approximately 30K documents, and its associated (denormalized) concept index, and merged the two into a single new index with the concept map flattened into the document as described above. I ran 5 concept queries, first as a TermQuery (with a custom sort on the concept score field) and then as a PayloadTermQuery, 5 times each, discarding the first query (to eliminate cache warmup overheads), and averaged the wallclock elapsed times for each query. Here are the results:

Query Term#-resultsTermQuery (ms)PayloadTermQuery (ms)
2800541460.251.5
2790981390.251.75
5348177500.757.0
2793084500.51.75
2800232500.50.75

So it appears that on average (excluding outliers), PayloadTermQuery calls are approximately 3-5 times slower than equivalent TermQuery calls. But they do offer a smaller disk (and consequently OS cache) footprint and a simpler programming model, so it remains to be seen if this makes sense for us to use.

Update: 2010-10-11

The situation changes when you factor in the actual document retrieval (ie, page through the ScoreDoc array and get the Documents from the searcher using searcher.doc(ScoreDoc.doc)). It appears that the PayloadTermQuery approach is consistently faster, but not significantly so.

Query Term#-resultsTermQuery (ms)PayloadTermQuery (ms)
28005414612.59,25
27909813910.06.75
5348177509.259.0
2793084506.56.0
2800232505.754.5

Posted by Sujit Pal at 4:20 PM


:

[Elasticsearch] [링크 펌]Custom Similarity...

Elastic/Elasticsearch 2013. 7. 15. 23:12

http://elasticsearch-users.115913.n3.nabble.com/custom-similarity-setting-does-not-work-with-version-0-20-2-td4029500.html



ElasticSearch Users

custom similarity setting does not work with version 0.20.2

13 messages Options Options 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 3:47pm

custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
benjamin leviant
Reply | Threaded | More     star
Feb 08, 2013; 5:33pm

Re: custom similarity setting does not work with version 0.20.2

benjamin leviant
13 posts
Hello,

you should try this command :
curl -XPOST 'http://host:port/tweeter/' -d '
{
    "settings": {
        "index": {
            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 5:40pm

Re: custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
That does not work too. I tried it before.

On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
benjamin leviant
Reply | Threaded | More     star
Feb 08, 2013; 6:42pm

Re: custom similarity setting does not work with version 0.20.2

benjamin leviant
13 posts
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 7:00pm

Re: custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.

On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
benjamin leviant
Reply | Threaded | More     star
Feb 08, 2013; 7:27pm

Re: custom similarity setting does not work with version 0.20.2

benjamin leviant
13 posts
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 8:15pm

Re: custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207



On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
benjamin leviant
Reply | Threaded | More     star
Feb 08, 2013; 8:43pm

Re: custom similarity setting does not work with version 0.20.2

benjamin leviant
13 posts
Your implementation looks good.

But to get payload working in elasticsearch, a custom similarity is not enough.

You need also to implement several custom elements :
- a token filter : to index payload values
- a query parser : to score using payload values

Do you have all these elements working ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 12:15 PM, Mustafa Sener <[hidden email]> wrote:
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207




On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 9:03pm

Re: custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
I have token filter. But not query parser. I use more like this query directly. I will not use token payloads on query, I want to use token payloads stored by filter on index time. Do I still need a query parser even if I use more like this query? 

On Fri, Feb 8, 2013 at 1:43 PM, benjamin leviant <[hidden email]> wrote:
Your implementation looks good.

But to get payload working in elasticsearch, a custom similarity is not enough.

You need also to implement several custom elements :
- a token filter : to index payload values
- a query parser : to score using payload values

Do you have all these elements working ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 12:15 PM, Mustafa Sener <[hidden email]> wrote:
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207




On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
benjamin leviant
Reply | Threaded | More     star
Feb 08, 2013; 10:20pm

Re: custom similarity setting does not work with version 0.20.2

benjamin leviant
13 posts
It is not working, because the "more like this" query do not call the scorePayload method to score the results.

To use payloads, you need a custom query parser with a query having a scoring method that take in account payloads.

You can see documentation about lucene PayloadTermQuery and PayloadNearQuery.

Regards

Benjamin


On Fri, Feb 8, 2013 at 1:03 PM, Mustafa Sener <[hidden email]> wrote:
I have token filter. But not query parser. I use more like this query directly. I will not use token payloads on query, I want to use token payloads stored by filter on index time. Do I still need a query parser even if I use more like this query? 

On Fri, Feb 8, 2013 at 1:43 PM, benjamin leviant <[hidden email]> wrote:
Your implementation looks good.

But to get payload working in elasticsearch, a custom similarity is not enough.

You need also to implement several custom elements :
- a token filter : to index payload values
- a query parser : to score using payload values

Do you have all these elements working ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 12:15 PM, Mustafa Sener <[hidden email]> wrote:
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207




On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 10:32pm

Re: custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
Ok,
Thanks for your help. It was very beneficial for me.

Regards...

On Fri, Feb 8, 2013 at 3:20 PM, benjamin leviant <[hidden email]> wrote:
It is not working, because the "more like this" query do not call the scorePayload method to score the results.

To use payloads, you need a custom query parser with a query having a scoring method that take in account payloads.

You can see documentation about lucene PayloadTermQuery and PayloadNearQuery.

Regards

Benjamin


On Fri, Feb 8, 2013 at 1:03 PM, Mustafa Sener <[hidden email]> wrote:
I have token filter. But not query parser. I use more like this query directly. I will not use token payloads on query, I want to use token payloads stored by filter on index time. Do I still need a query parser even if I use more like this query? 

On Fri, Feb 8, 2013 at 1:43 PM, benjamin leviant <[hidden email]> wrote:
Your implementation looks good.

But to get payload working in elasticsearch, a custom similarity is not enough.

You need also to implement several custom elements :
- a token filter : to index payload values
- a query parser : to score using payload values

Do you have all these elements working ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 12:15 PM, Mustafa Sener <[hidden email]> wrote:
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207




On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Mustafa Sener
Reply | Threaded | More     star
Feb 08, 2013; 10:51pm

Re: custom similarity setting does not work with version 0.20.2

Mustafa Sener
77 posts
The best option here to use morelikethis query is modify it to take payloads into account. I think I can do this by sub-classing payload query and multiply tf value used by query by payload of that term. What do you think about this design? Actually I need a categorizer which when I enter a text will return best matching categories based on predefined terms and payloads for categories. I select more like this query for this purpose. Payloads are important because I will assign negative payloads to negative samples or terms.

On Fri, Feb 8, 2013 at 3:32 PM, Mustafa Sener <[hidden email]> wrote:
Ok,
Thanks for your help. It was very beneficial for me.

Regards...


On Fri, Feb 8, 2013 at 3:20 PM, benjamin leviant <[hidden email]> wrote:
It is not working, because the "more like this" query do not call the scorePayload method to score the results.

To use payloads, you need a custom query parser with a query having a scoring method that take in account payloads.

You can see documentation about lucene PayloadTermQuery and PayloadNearQuery.

Regards

Benjamin


On Fri, Feb 8, 2013 at 1:03 PM, Mustafa Sener <[hidden email]> wrote:
I have token filter. But not query parser. I use more like this query directly. I will not use token payloads on query, I want to use token payloads stored by filter on index time. Do I still need a query parser even if I use more like this query? 

On Fri, Feb 8, 2013 at 1:43 PM, benjamin leviant <[hidden email]> wrote:
Your implementation looks good.

But to get payload working in elasticsearch, a custom similarity is not enough.

You need also to implement several custom elements :
- a token filter : to index payload values
- a query parser : to score using payload values

Do you have all these elements working ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 12:15 PM, Mustafa Sener <[hidden email]> wrote:
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207




On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener




-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
benjamin leviant
Reply | Threaded | More     star
Feb 09, 2013; 12:41am

Re: custom similarity setting does not work with version 0.20.2

benjamin leviant
13 posts
Yes you can try by subclassing morelikethis query to customize its scoring logic. 

Sorry, but I cannot confirm you it will be work. 

Please, give us update on this.

Regards


On Fri, Feb 8, 2013 at 2:51 PM, Mustafa Sener <[hidden email]> wrote:
The best option here to use morelikethis query is modify it to take payloads into account. I think I can do this by sub-classing payload query and multiply tf value used by query by payload of that term. What do you think about this design? Actually I need a categorizer which when I enter a text will return best matching categories based on predefined terms and payloads for categories. I select more like this query for this purpose. Payloads are important because I will assign negative payloads to negative samples or terms.


On Fri, Feb 8, 2013 at 3:32 PM, Mustafa Sener <[hidden email]> wrote:
Ok,
Thanks for your help. It was very beneficial for me.

Regards...


On Fri, Feb 8, 2013 at 3:20 PM, benjamin leviant <[hidden email]> wrote:
It is not working, because the "more like this" query do not call the scorePayload method to score the results.

To use payloads, you need a custom query parser with a query having a scoring method that take in account payloads.

You can see documentation about lucene PayloadTermQuery and PayloadNearQuery.

Regards

Benjamin


On Fri, Feb 8, 2013 at 1:03 PM, Mustafa Sener <[hidden email]> wrote:
I have token filter. But not query parser. I use more like this query directly. I will not use token payloads on query, I want to use token payloads stored by filter on index time. Do I still need a query parser even if I use more like this query?

On Fri, Feb 8, 2013 at 1:43 PM, benjamin leviant <[hidden email]> wrote:
Your implementation looks good.

But to get payload working in elasticsearch, a custom similarity is not enough.

You need also to implement several custom elements :
- a token filter : to index payload values
- a query parser : to score using payload values

Do you have all these elements working ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 12:15 PM, Mustafa Sener <[hidden email]> wrote:
I use a FloatingPayload filter to create payloads for each term which is separated by '|' character (school|5.8). Then I use following similarity 

https://gist.github.com/anonymous/4738207




On Fri, Feb 8, 2013 at 12:27 PM, benjamin leviant <[hidden email]> wrote:
I use a custom similarity on Elasticsearch 0.20.1 and it work perfectly well.

If you dont have any error message when you create your index, then your custom similarity should be enable for search and index operations on this index.

May be your problem is caused by the way your similarity provider is implemented. Can you post the code ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 11:00 AM, Mustafa Sener <[hidden email]> wrote:
I have no error messages. It seems that this setting is different before version 0.20. Since there is no documentation about this, I cannot configure it properly.


On Fri, Feb 8, 2013 at 11:42 AM, benjamin leviant <[hidden email]> wrote:
Hi,

What is your the error message ?

How do you get your custom class loaded ?

Regards

Benjamin

On Fri, Feb 8, 2013 at 9:40 AM, Mustafa Sener <[hidden email]> wrote:
That does not work too. I tried it before.


On Fri, Feb 8, 2013 at 10:33 AM, benjamin leviant <[hidden email]> wrote:
Hello,

you should try this command :
curl -XPOST 'http://host:port
/tweeter/' -d '
{
    "settings": {
        "index": {

            "similarity": {
                "index": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                },
                "search": {
                    "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
                }
            }
        }
    }
}
I hope it can help you.
Regards

Benjamin


On Fri, Feb 8, 2013 at 7:47 AM, Mustafa Sener <[hidden email]> wrote:
Hi,
I tried following configurations for my custom similarity provider but none of them worked with version 0.20. Can anyone give me some information about this setting for version 0.20.2? Any sample usages will be enough for me

curl -XPOST 'http://host:port/tweeter/' -d '
{
  "settings": {
    "similarity": {
      "index": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      },
      "search": {
        "type": "org.elasticsearch.index.similarity.CustomSimilarityProvider"
      }
    }
  }
}


 index.similarity.index.type
 index.similarity.search.type

None of these works.


Thanks...


-- 
Mustafa Sener





-- 
Mustafa Sener
www.ifountain.com

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 



-- 
Mustafa Sener




-- 
Mustafa Sener

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Powered by NabbleEdit this page


:
◀ PREV : [1] : NEXT ▶

티스토리툴바