Início > bigdata, JAVA, watson explorer, wex > Finding bottlenecks at Watson Explorer queries

Finding bottlenecks at Watson Explorer queries

If you are having problem with some Watson Explorer query, an excellent way to find bottlenecks is to perform the query with Debug and Profile options enabled, it will help you to find where exactly you have problems.

Usually, when you perform a query at WEX, you call some URL like the following (in my case port is 7205, MY_COLLECTION can be a shard, for example MY_COLLECTION_1_1):

<SERVER>:<PORT>/search?collection=MY_COLLECTION&query-xml=<%3fxml version%3d”1.0″ encoding%3d”UTF-8″%3f><operator logic%3d”and”%2f>&num=1&max=1&binning-mode=normal&start=0&show-duplicates=1&doc-axl=<%3fxml version%3d”1.0″ encoding%3d”UTF-8″%3f><document key-hash%3d”{vse%3adoc-hash()}”%2f>&binning-config=<%3fxml version%3d”1.0″ encoding%3d”UTF-8″%3f><binning-sets><binning-set bs-id%3d”VENDOR” logic%3d”or” max-bins%3d”8″ select%3d”%24VENDOR”%2f><binning-set bs-id%3d”REVENUE_USD_FACET” logic%3d”or” max-bins%3d”11″ select%3d”%24REVENUE_USD_FACET”%2f>……………field%3d”SERVICE_AREA”><field-to name%3d”SERVICE_AREA”%2f><%2ffield-map><field-map field%3d”MAX_IGS_REV_OM_BRAND_CD”><field-to name%3d”MAX_IGS_REV_OM_BRAND_CD”%2f><%2ffield-map><field-map field%3d”EMAIL_SENT”><field-to name%3d”EMAIL_SENT”%2f><%2ffield-map><field-map field%3d”REVENUE_USD_FACET”><field-to name%3d”REVENUE_USD_FACET”%2f><%2ffield-map><field-map field%3d”REVENUE”><field-to name%3d”REVENUE”%2f><%2ffield-map><field-map field%3d”CLIENT_NAME”><field-to name%3d”CLIENT_NAME”%2f><%2ffield-map><%2ffield-mapping>&sort-keys=1&score=1&shingles=0&summarize=0&gen-key=0&cache-data=0&force-binning=1&output-acls=1

If you don’t have IDEA about HOW to get the query that your Application is doing, you can enable Debug at your collection. Go to WEX console, under Configuration -> Searching -> Debugging and enable Query Logging.

Selection_355

When saved, it will start to generate log in a file called queries.log, under you collection folder, some place like:

/opt/IBM/dataexplorer/WEX-11/Engine/data/search-collections/YYY/MY_COLLECTION/crawl1/

You can check it at WEX console, under your collection configuration, tab META, field Filebase.

Ok, now, if you call this URL from your browser, appending “&debug=1&profile=1″ to the URL, you will got a XML file. Download it and lets analyze. For our case, see this:

<xpath-performances>
<xpath-performance xpath=”($FIELD_X) = ‘GBS – No’ or ($FIELD_X) = ‘GBS – Yes'” slow-ms=”10295″ n-slow=”192000″ n-fast=”0″ n-direct=”0″ n-hashes=”1″ />
</xpath-performances>

THIS tell me that JUST in order to get the field FIELD_X, I’m having slow! (I’m my case it is because my Field its an Array)

So, probably I have a problem with this field, that can be a lot, for example:
1- Null values (see my other posts)
2- Its an array to index
3- Its a long text field
4- You have a lot of possible statements using it (OR, AND, WHERE, etc)

With this information, you can go to next step, that is find a way to change the field and make it work better.

Important: I tested this with Watson Explorer 9, 10 and 11. Running at Linux Machines.

Enjoy!

  1. Nenhum comentário ainda.
  1. No trackbacks yet.

Deixe uma resposta

Preencha os seus dados abaixo ou clique em um ícone para log in:

Logotipo do WordPress.com

Você está comentando utilizando sua conta WordPress.com. Sair / Alterar )

Imagem do Twitter

Você está comentando utilizando sua conta Twitter. Sair / Alterar )

Foto do Facebook

Você está comentando utilizando sua conta Facebook. Sair / Alterar )

Foto do Google+

Você está comentando utilizando sua conta Google+. Sair / Alterar )

Conectando a %s

%d blogueiros gostam disto: