Metrics with eland

Use the eland library to generate search metrics based on data in an Elasticsearch index. All indices and transforms should have already been created and run before this notebook can be used. We operate on the two main indices: events in ecs-search-metrics and the post-transform query-level metrics in ecs-search-metrics_transform_queryid.


In [1]:
%matplotlib inline

import eland as el
import numpy as np

In [2]:
ES_URL = 'http://localhost:9200/'

Data Loading and Preparation

Event Index

The ecs-search-metrics index contains the raw behavioural events: query, page and click.


In [3]:
df = el.read_es(ES_URL, 'ecs-search-metrics')

In [4]:
df.dtypes


Out[4]:
@timestamp                                    datetime64[ns]
SearchMetrics.click.result.id                         object
SearchMetrics.click.result.rank                        int64
SearchMetrics.click.result.reciprocal_rank           float64
SearchMetrics.query.id                                object
SearchMetrics.query.page                               int64
SearchMetrics.query.value                             object
SearchMetrics.results.ids                             object
SearchMetrics.results.size                             int64
SearchMetrics.results.total                            int64
SearchMetricsSimulation.ab.experiment                 object
SearchMetricsSimulation.ab.variant                    object
SearchMetricsSimulation.page_name                     object
ecs.version                                           object
event.action                                          object
event.dataset                                         object
event.duration                                         int64
event.id                                              object
source.geo.city_name                                  object
source.geo.country_iso_code                           object
source.geo.location                                   object
source.user.id                                        object
dtype: object

In [5]:
print(df.info_es())


es_index_pattern: ecs-search-metrics
Index:
 es_index_field: _id
 is_source_field: False
Mappings:
 capabilities:
                                                                         es_field_name  is_source   es_dtype es_date_format        pd_dtype  is_searchable  is_aggregatable  is_scripted                  aggregatable_es_field_name
@timestamp                                                                  @timestamp       True       date           None  datetime64[ns]           True             True        False                                  @timestamp
SearchMetrics.click.result.id                            SearchMetrics.click.result.id       True    keyword           None          object           True             True        False               SearchMetrics.click.result.id
SearchMetrics.click.result.rank                        SearchMetrics.click.result.rank       True       long           None           int64           True             True        False             SearchMetrics.click.result.rank
SearchMetrics.click.result.reciprocal_rank  SearchMetrics.click.result.reciprocal_rank       True      float           None         float64           True             True        False  SearchMetrics.click.result.reciprocal_rank
SearchMetrics.query.id                                          SearchMetrics.query.id       True    keyword           None          object           True             True        False                      SearchMetrics.query.id
SearchMetrics.query.page                                      SearchMetrics.query.page       True       long           None           int64           True             True        False                    SearchMetrics.query.page
SearchMetrics.query.value                                    SearchMetrics.query.value       True    keyword           None          object           True             True        False                   SearchMetrics.query.value
SearchMetrics.results.ids                                    SearchMetrics.results.ids       True    keyword           None          object           True             True        False                   SearchMetrics.results.ids
SearchMetrics.results.size                                  SearchMetrics.results.size       True       long           None           int64           True             True        False                  SearchMetrics.results.size
SearchMetrics.results.total                                SearchMetrics.results.total       True       long           None           int64           True             True        False                 SearchMetrics.results.total
SearchMetricsSimulation.ab.experiment            SearchMetricsSimulation.ab.experiment       True    keyword           None          object           True             True        False       SearchMetricsSimulation.ab.experiment
SearchMetricsSimulation.ab.variant                  SearchMetricsSimulation.ab.variant       True    keyword           None          object           True             True        False          SearchMetricsSimulation.ab.variant
SearchMetricsSimulation.page_name                    SearchMetricsSimulation.page_name       True    keyword           None          object           True             True        False           SearchMetricsSimulation.page_name
ecs.version                                                                ecs.version       True    keyword           None          object           True             True        False                                 ecs.version
event.action                                                              event.action       True    keyword           None          object           True             True        False                                event.action
event.dataset                                                            event.dataset       True    keyword           None          object           True             True        False                               event.dataset
event.duration                                                          event.duration       True       long           None           int64           True             True        False                              event.duration
event.id                                                                      event.id       True    keyword           None          object           True             True        False                                    event.id
source.geo.city_name                                              source.geo.city_name       True    keyword           None          object           True             True        False                        source.geo.city_name
source.geo.country_iso_code                                source.geo.country_iso_code       True    keyword           None          object           True             True        False                 source.geo.country_iso_code
source.geo.location                                                source.geo.location       True  geo_point           None          object           True             True        False                         source.geo.location
source.user.id                                                          source.user.id       True    keyword           None          object           True             True        False                              source.user.id
Operations:
 tasks: []
 size: None
 sort_params: None
 _source: ['@timestamp', 'SearchMetrics.click.result.id', 'SearchMetrics.click.result.rank', 'SearchMetrics.click.result.reciprocal_rank', 'SearchMetrics.query.id', 'SearchMetrics.query.page', 'SearchMetrics.query.value', 'SearchMetrics.results.ids', 'SearchMetrics.results.size', 'SearchMetrics.results.total', 'SearchMetricsSimulation.ab.experiment', 'SearchMetricsSimulation.ab.variant', 'SearchMetricsSimulation.page_name', 'ecs.version', 'event.action', 'event.dataset', 'event.duration', 'event.id', 'source.geo.city_name', 'source.geo.country_iso_code', 'source.geo.location', 'source.user.id']
 body: {}
 post_processing: []


In [6]:
df.head()


Out[6]:
@timestamp SearchMetrics.click.result.id SearchMetrics.click.result.rank SearchMetrics.click.result.reciprocal_rank SearchMetrics.query.id SearchMetrics.query.page SearchMetrics.query.value SearchMetrics.results.ids SearchMetrics.results.size SearchMetrics.results.total ... SearchMetricsSimulation.page_name ecs.version event.action event.dataset event.duration event.id source.geo.city_name source.geo.country_iso_code source.geo.location source.user.id
FbCZdHIBV286dKaWk0Zt 2019-11-15 23:39:37+00:00 NaN NaN NaN 23bd7c41-e138-4a75-b6f0-c58750d0ee17 1 By Very hospital possible base [17, 86, 20, 48, 30, 51, 52, 62, 7, 60] 10.0 63.0 ... product_search_page 1.6.0-dev SearchMetrics.query SearchMetrics.query 254000000.0 23bd7c41-e138-4a75-b6f0-c58750d0ee17 Soroca MD 48.15659,28.28489 0
FrCZdHIBV286dKaWk0aa 2019-11-15 23:39:37+00:00 NaN NaN NaN 23bd7c41-e138-4a75-b6f0-c58750d0ee17 2 NaN [55, 56, 90, 26, 54, 15, 64, 6, 27, 97] 10.0 NaN ... NaN 1.6.0-dev SearchMetrics.page SearchMetrics.page 274.0 279ace95-46cf-438e-a31e-bf82daa8f64b NaN NaN NaN NaN
F7CZdHIBV286dKaWk0bR 2019-11-15 23:40:34+00:00 55 11.0 0.090909 23bd7c41-e138-4a75-b6f0-c58750d0ee17 2 NaN NaN NaN NaN ... NaN 1.6.0-dev SearchMetrics.click SearchMetrics.click NaN 87deadce-3e18-455f-9506-7c89aa1dba16 NaN NaN NaN NaN
GLCZdHIBV286dKaWk0b- 2019-11-15 23:40:18+00:00 15 16.0 0.062500 23bd7c41-e138-4a75-b6f0-c58750d0ee17 2 NaN NaN NaN NaN ... NaN 1.6.0-dev SearchMetrics.click SearchMetrics.click NaN 7344dede-169e-4d7f-aa93-221a2a32eac2 NaN NaN NaN NaN
GbCZdHIBV286dKaWlEYs 2019-11-15 16:39:07+00:00 NaN NaN NaN 9b9e7d08-73bb-414c-9306-37f6bbb97943 1 Coach condition history window [56, 26, 18, 76, 9, 16, 79, 61, 12, 57] 10.0 77.0 ... home_page_onebox 1.6.0-dev SearchMetrics.query SearchMetrics.query 44000000.0 9b9e7d08-73bb-414c-9306-37f6bbb97943 Karbala IQ 32.61603,44.02488 1

5 rows × 22 columns

What is the distribution of ranks of results clicked on?


In [7]:
df['SearchMetrics.click.result.rank'].describe()


Out[7]:
SearchMetrics.click.result.rank
count 84.000000
mean 8.440476
std 5.423387
min 1.000000
25% 4.000000
50% 8.000000
75% 13.000000
max 20.000000

In [8]:
df['SearchMetrics.click.result.rank'].hist()


Out[8]:
<matplotlib.axes._subplots.AxesSubplot at 0x11ec9d7d0>

How many users are in the dataset?


In [9]:
df['source.user.id'].nunique()


Out[9]:
10

How many of each event type?


In [10]:
df['event.action'].value_counts()


Out[10]:
SearchMetrics.click    84
SearchMetrics.query    32
SearchMetrics.page     12
Name: event.action, dtype: int64

Split dataset into two dataframes based on action type.


In [11]:
df_queries = df[df['event.action'] == 'SearchMetrics.query']
df_pages = df[df['event.action'] == 'SearchMetrics.page']
df_clicks = df[df['event.action'] == 'SearchMetrics.click']

What is the distribution of search result sizes in query events?


In [12]:
df_queries[['SearchMetrics.results.size']].hist(figsize=[10,5], bins=10)


Out[12]:
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x120e33e50>]],
      dtype=object)

Metrics Index

The ecs-search-metrics_transform_queryid index is used to store per-query metrics.


In [13]:
df_tf_query = el.read_es(ES_URL, 'ecs-search-metrics_transform_queryid')

In [14]:
df_tf_query.head()


Out[14]:
SearchMetrics.query.id metrics.clicks.count metrics.clicks.exist_at_10 metrics.clicks.exist_at_3 metrics.clicks.max_page_clicked metrics.clicks.max_reciprocal_rank metrics.clicks.mean_reciprocal_rank metrics.clicks.num_above_fold metrics.clicks.num_below_fold metrics.clicks.time_to_first_click ... query_event.SearchMetricsSimulation.ab.variant query_event.SearchMetricsSimulation.page_name query_event.event.action query_event.event.dataset query_event.event.duration query_event.event.id query_event.source.geo.city_name query_event.source.geo.country_iso_code query_event.source.geo.location query_event.source.user.id
MIOMlGA6bpeYgWvvLXEuaqgAAAAAAAAA 065ef3cb-4e2d-422e-8459-2b0a17353cca 7.0 True True 2.0 0.5 0.145088 3 4 14000.0 ... a user_search_page SearchMetrics.query SearchMetrics.query 186000000 065ef3cb-4e2d-422e-8459-2b0a17353cca Roi Et TH 16.0567,103.65309 5
MBho1dYeVG3gUCDt7i7yJqQAAAAAAAAA 04f65713-25d6-4b3c-9d03-89527e61a5c2 0.0 False False NaN NaN NaN 0 0 NaN ... a user_search_page SearchMetrics.query SearchMetrics.query 259000000 04f65713-25d6-4b3c-9d03-89527e61a5c2 Bua Yai TH 15.58552,102.42587 5
MfqN0amSAXcx9yIiEMtz0vAAAAAAAAAA 1b84bb80-2104-4155-89c6-d7c525d1d24b 0.0 False False NaN NaN NaN 0 0 NaN ... control user_search_page SearchMetrics.query SearchMetrics.query 48000000 1b84bb80-2104-4155-89c6-d7c525d1d24b Karbala IQ 32.61603,44.02488 1
MTEjDeSBb2BdFmpQc3RPCwEAAAAAAAAA 19e4da1c-8910-43c4-a9e1-34b4686efdea 2.0 True True 1.0 0.5 0.333333 2 0 33000.0 ... control user_search_page SearchMetrics.query SearchMetrics.query 236000000 19e4da1c-8910-43c4-a9e1-34b4686efdea Al Jubayl SA 27.0174,49.62251 7
MkU4hXtof9W_dNgWe3FuqBgAAAAAAAAA 22450eda-064d-4951-8d9c-d259200376bd 0.0 False False NaN NaN NaN 0 0 NaN ... none user_search_page SearchMetrics.query SearchMetrics.query 274000000 22450eda-064d-4951-8d9c-d259200376bd Karbala IQ 32.61603,44.02488 4

5 rows × 30 columns

What are the distributions of the numeric fields?


In [15]:
df_tf_query.select_dtypes(include=[np.number])\
  .drop(['query_event.SearchMetrics.query.page'], axis=1)\
  .hist(figsize=[17,11], bins=10)


Out[15]:
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x120f59090>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x12106a210>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x12109d890>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x1210cef10>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1211125d0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x11eb90390>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x120f42710>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x12113efd0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1211b1050>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x1211e57d0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x12125f410>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x121295a90>]],
      dtype=object)

Metrics

We're going to start with some basic query metrics [1] (no session metrics):

  • Zero Result Rate: The fraction of queries for which no results were returned.
  • Abandonment Rate^: The fraction of queries for which no results were clicked on.
  • Clicks per Query^: The mean number of results that are clicked for each query.
  • Max Reciprocal Rank^†: The mean value of 1/r, where r is the rank of the highest ranked result clicked on.
  • Mean Reciprocal Rank^†: The mean value of Σ 1/rᵢ, summing over the ranks rᵢ of all clicks for each query.
  • Time to First Click^†: The mean time from query being issued until first click on any result.
  • Time to Last Click^†: The mean time from query being issued until last click on any result.

^ When computing the metrics marked with ^, we exclude queries with no results to avoid conflating these measures with zero result rate.
† When computing the metrics marked with †, we exclude queries with no clicks to avoid conflating these measures with abandonment rate.

[1] F. Radlinski, M. Kurup, T. Joachims. How Does Clickthrough Data Reflect Retrieval Quality?. CIKM '08, 2008.

Given the above definitions, build datasets to base metrics off of.


In [16]:
# queries that have no results
df_tf_query_without_results = df_tf_query[df_tf_query['query_event.SearchMetrics.results.size'] == 0]

# queries that have results
df_tf_query_with_results = df_tf_query[df_tf_query['query_event.SearchMetrics.results.size'] > 0]

# queries that have results but no clicks
df_tf_query_without_clicks = df_tf_query_with_results[df_tf_query_with_results['metrics.clicks.count'] == 0]

# queries that have results and clicks
df_tf_query_with_clicks = df_tf_query_with_results[df_tf_query_with_results['metrics.clicks.count'] > 0]

Provide basic counts for all datasets.


In [17]:
num_queries = df_tf_query.shape[0]
num_queries_without_results = df_tf_query_without_results.shape[0]
num_queries_with_results = df_tf_query_with_results.shape[0]
num_queries_without_clicks = df_tf_query_without_clicks.shape[0]
num_queries_with_clicks = df_tf_query_with_clicks.shape[0]

Zero Result Rate

Lower is better.
Best possible value: 0.0%


In [18]:
zero_result_rate = num_queries_without_results / num_queries * 100

print(f"Zero result rate: {round(zero_result_rate, 2)}%")


Zero result rate: 9.38%

Abandonment Rate

Lower is better.
Best possible value: 0.0%


In [19]:
abandonment_rate = num_queries_without_clicks / num_queries * 100

print(f"Abandonment rate: {round(abandonment_rate, 2)}%")


Abandonment rate: 12.5%

Clicks per Query

Lower is better.
Best possible value: 1.0


In [20]:
mean_clicks_per_query = df_tf_query_with_results['metrics.clicks.count'].mean()

print(f"Clicks per Query: {round(mean_clicks_per_query, 2)}")


Clicks per Query: 2.9

Click Through Rate at Position 3 (CTR@3)

Higher is better.
Best possible value: 1.0


In [21]:
num_queries_with_clicks_at_3 = df_tf_query_with_clicks[df_tf_query_with_clicks['metrics.clicks.exist_at_3'] == True].shape[0]
ctr_at_3 = num_queries_with_clicks_at_3 / num_queries_with_clicks

print(f"CTR@3: {round(ctr_at_3, 2)}")


CTR@3: 0.56

Max Reciprocal Rank

Higher is better.
Best possible value: 1.0


In [22]:
max_reciprocal_rank = df_tf_query_with_clicks['metrics.clicks.max_reciprocal_rank'].mean()

print(f"Max Reciprocal Rank: {round(max_reciprocal_rank, 2)}")


Max Reciprocal Rank: 0.46

Mean Reciprocal Rank

Higher is better.
Best possible value: 1.0


In [23]:
mean_mean_reciprocal_rank = df_tf_query_with_clicks['metrics.clicks.mean_reciprocal_rank'].mean()

print(f"Mean Per-Query Mean Reciprocal Rank: {round(mean_mean_reciprocal_rank, 2)}")


Mean Per-Query Mean Reciprocal Rank: 0.28

Time to First Click

Lower is better.
Best possible value: 0.0


In [24]:
time_to_first_click = df_tf_query_with_clicks['metrics.clicks.time_to_first_click'].mean()

print(f"Time to First Click: {round(time_to_first_click / 1000, 2)} seconds")


Time to First Click: 21.2 seconds

Time to Last Click

Higher is better (infers many interesting results).


In [25]:
time_to_last_click = df_tf_query_with_clicks['metrics.clicks.time_to_last_click'].mean()

print(f"Time to Last Click: {round(time_to_last_click / 1000, 2)} seconds")


Time to Last Click: 50.52 seconds

In [ ]: