Archive

Posts Tagged ‘debug’

Finding bottlenecks at Watson Explorer queries

If you are having problem with some Watson Explorer query, an excellent way to find bottlenecks is to perform the query with Debug and Profile options enabled, it will help you to find where exactly you have problems.

Usually, when you perform a query at WEX, you call some URL like the following (in my case port is 7205, MY_COLLECTION can be a shard, for example MY_COLLECTION_1_1):

<SERVER>:<PORT>/search?collection=MY_COLLECTION&query-xml=<%3fxml version%3d”1.0″ encoding%3d”UTF-8″%3f><operator logic%3d”and”%2f>&num=1&max=1&binning-mode=normal&start=0&show-duplicates=1&doc-axl=<%3fxml version%3d”1.0″ encoding%3d”UTF-8″%3f><document key-hash%3d”{vse%3adoc-hash()}”%2f>&binning-config=<%3fxml version%3d”1.0″ encoding%3d”UTF-8″%3f><binning-sets><binning-set bs-id%3d”VENDOR” logic%3d”or” max-bins%3d”8″ select%3d”%24VENDOR”%2f><binning-set bs-id%3d”REVENUE_USD_FACET” logic%3d”or” max-bins%3d”11″ select%3d”%24REVENUE_USD_FACET”%2f>……………field%3d”SERVICE_AREA”><field-to name%3d”SERVICE_AREA”%2f><%2ffield-map><field-map field%3d”MAX_IGS_REV_OM_BRAND_CD”><field-to name%3d”MAX_IGS_REV_OM_BRAND_CD”%2f><%2ffield-map><field-map field%3d”EMAIL_SENT”><field-to name%3d”EMAIL_SENT”%2f><%2ffield-map><field-map field%3d”REVENUE_USD_FACET”><field-to name%3d”REVENUE_USD_FACET”%2f><%2ffield-map><field-map field%3d”REVENUE”><field-to name%3d”REVENUE”%2f><%2ffield-map><field-map field%3d”CLIENT_NAME”><field-to name%3d”CLIENT_NAME”%2f><%2ffield-map><%2ffield-mapping>&sort-keys=1&score=1&shingles=0&summarize=0&gen-key=0&cache-data=0&force-binning=1&output-acls=1

If you don’t have IDEA about HOW to get the query that your Application is doing, you can enable Debug at your collection. Go to WEX console, under Configuration -> Searching -> Debugging and enable Query Logging.

Selection_355

When saved, it will start to generate log in a file called queries.log, under you collection folder, some place like:

/opt/IBM/dataexplorer/WEX-11/Engine/data/search-collections/YYY/MY_COLLECTION/crawl1/

You can check it at WEX console, under your collection configuration, tab META, field Filebase.

Ok, now, if you call this URL from your browser, appending “&debug=1&profile=1″ to the URL, you will got a XML file. Download it and lets analyze. For our case, see this:

<xpath-performances>
<xpath-performance xpath=”($FIELD_X) = ‘GBS – No’ or ($FIELD_X) = ‘GBS – Yes'” slow-ms=”10295″ n-slow=”192000″ n-fast=”0″ n-direct=”0″ n-hashes=”1″ />
</xpath-performances>

THIS tell me that JUST in order to get the field FIELD_X, I’m having slow! (I’m my case it is because my Field its an Array)

So, probably I have a problem with this field, that can be a lot, for example:
1- Null values (see my other posts)
2- Its an array to index
3- Its a long text field
4- You have a lot of possible statements using it (OR, AND, WHERE, etc)

With this information, you can go to next step, that is find a way to change the field and make it work better.

Important: I tested this with Watson Explorer 9, 10 and 11. Running at Linux Machines.

Enjoy!

Watson Explorer performance decrease with null values

Working with Watson Explorer (WEX) we saw that the search performance decrease a lot when you have null values for some field/facet. (Our WEX release at this moment is 11, we run at Linux machines and our application was written in Java, using BigIndex to index and search. (Also have pure REST version of our application in test and the problem still happen)).

For example: lets suppose that you have a facet called VENDOR in an entity called Product. Suppose that you have 5 millions Products indexed and for some of then you have nulls, in my case 2 Millions have NULL values for VENDOR field.

In this case (and similar ones), we notice a performance decrease in searches. We start to see problems when the relation of null are greater than 20%.

In order to solve the problem we have 2 options:

1- One technique we’ve used for certain dimensions is to always ensure non-null values in the index — so at index time, we either coalesce in our SQL pulls from DB2 or do transformation after ingestion to replace nulls with some predefined value. In our case we use the literal string “(no value available)”.  It: a) ensures non-null values, b) is fairly meaningful to users, and c) gives users a way to actually filter on those records if needed.

2- For some FIELDS, we can not add another values, must leave null (Business reasons) and must not show null option to user select in the facet. In this case, in the moment of search we append boolean($FIELD) to the query. For example:

General example of a slow facet query:
$VENDOR=’IBM or $VENDOR=’PEPSICO’
Rewritten query that solve the slowness:
boolean($VENDOR) and ($VENDOR=’IBM or $VENDOR=’PEPSICO’)
This way, it ignore the null values when searching and it will be very fast.
Maybe this is not the final solution and for newer WEX versions it will handle better with null values, but, this was the solution for us.
Enjoy!

Debugando aplicativos no IOs (IPhone)

É possível debugar uma aplicação rodando no IPhone (ou no emulador) direto pelo Navegador (Safari). Para fazer isso, conecte seu dispositivo via porta USB em seu computador ou inicie o emulador. No Safari, va em Preferences, Advanced e mande exibir as opções de Desenvolvimento:

Selection_082Feito isso, caso queira debugar de seu dispositivo físico, abra o Safari e desabilite o provate Browsing e habilite o Web Inspector am advanced:

Selection_083 Selection_084Feito isso, no Safari de seu desktop, na opção Develop, você ira ver seu dispositivo e conseguir selecionar sua aplicação.

Enjoy!

Debugando aplicativos no Android

Poucas pessoas sabem, porém é possível debugar uma aplicação rodando no Android direto pelo Navegador (Chrome). Para fazer isso, conecte seu dispositivo via porta USB em seu computador, vá em Tools > Devices (ou algo assim). A janela abaixo irá abrir:

Selection_077Então, abra o aplicativo no seu celular que quer ver e clique em Inspect.

Selection_078Veja o resultado:

Selection_079Isso ajuda muito e é fácil de se fazer. Existem outras formas como utilizar o comando “adb logcat” que vem junto com a SDK do Android e esta disponível em platform-tools, porém, a mais simples e de fácil entendimento é esta que apresentei (em minha opinião).

Enjoy!

Resolvendo problemas de Debug RAD 7 e 7.5

Eventualmente notamos que a função de Debug no RAD (Rational Application Developer) para de funcionar. Existe uma lenda urbana que diz que alterando uma porta isso se resolve, e realmente, após fazer isso, meu Debug nunca mais parou de funcionar. Segue o procedimento:

Faça o seguinte procedimento para resolver o problema, porém, utilize outra porta que não seja a 7777. (Está é a default é que gera problemas).
No meu caso usei a 7778

The instructions seem to be a little out of date so here are my instructions.
1.    Log into your Integrated Solutions Console.  The default URL is http://localhost:9060/ibm/console/login.do.
2.    Navigate using the left side column to Servers –> Application Servers.
3.    Select the Application server you want to debug from the list of Application servers.
4.    Under the Configuration tab select the Debugging Service link which is near the bottom right in the Additional Properties section.
5.    Select the “Enable service at server startup” checkbox.  Change the JVM debug port to other ex: 7778.
6.    Press the Apply button.
7.    In the Messages box, which appeared at the top after pressing the Apply button, click on the Save link.
8.    Stop and start your Application Server.  It should now be running in Debug mode.
9.    In RAD go to the project for the web application you want to debug.

Enjoy!

 

Categorias:JAVA Tags:, , ,