Watson Explorer Content Analytics installation – Steps

I installed Watson Explorer Content Analytics several times in the last months, then, I created a simple Step by step that can be useful to who is starting to work with this. I’m sharing here, please, fell free to add any comment and help me to improve this how-to. Hope that can be useful to you if you are here :D.

Pre-Installation Checklist / Pre-requisites

The following steps should be performed to prepare the server for software installation. This step by step cover the installation in a Red Hat 7 machine. The following 32-bit libraries are required on Linux x86-64 systems:

• libstdc++33 (compat-libstdc++-33.i686 / libstdc++33-32bit)
• libstdc++ (libstdc++.i686 / libstdc++6-32bit)
• zlib (zlib.i686 / libz1-32bit / zlib-32bit)
• libXext (libXext.i686 / libXext6-32bit / xorg-x11-libXext-32bit)
• libXft (libXft.i686 / libXft2-32bit / xorg-x11-libs-32bit)
• libXi (libXi.i686 / libXi6-32bit / xorg-x11-libs-32bit)
• libXp (libXp.i686 / libXp6-32bit / xorg-x11-libXp-32bit)
• libXtst (libXtst.i686 / libXtst6-32bit / xorg-x11-libs-32bit)
• libXm (if thumbnails are needed)
• libXt (if thumbnails are needed)
• java-1.8.0-ibm-1.8.0.2.10-1jpp.7.el6.x86_64
• Httpd

The commands to install the libraries are:

1. yum install -y libstdc++.i686 libXext.i686 libXft.i686 libXi.i686 libXp.i686 libXtst.i686 libXt.i68
2. yum install -y zlib.i686 httpd
3. yum -y install apr apr-util boost-filesystem boost-iostreams boost-program-options boost-regex boost-serialization

• If SELinux is enabled, then it will need to be disabled during installation of WEX
  

• Ensure that the disk space has been allocated to the WEX installation path

• The default path is /opt/ibm/dataexplorer

• Ensure that the disk space for /tmp partition have sufficient space for the installation, at least 10 gb

If not assign the tmp to a directory that has extra space with the commands:

1. mkdir /opt/IBM/dataexplorer/tmp
2. export IATEMPDIR=/opt/IBM/dataexplorer/tmp

• Install a web server to run the Foundational Components -> On Linux install the latest version of Apache (not required for AC)

• Ensure that the LINUX subscription libraries are available for the yum command

• DB2 connect client configured

• Install Java – java-1.8.0-ibm-1.8.0.2.10-1jpp.7.el6.x86_64 (you can get the package from oracle also, make sure to add Java to the server path)
  

Users and Folders creation for WEX AE (must be perform with root level access)

1. Add installation account (esadmin) for server install.
   1. useradd esadmin
   2. create a password for the esadmin user (usually esadmin12)

3. Create Watson Directories
   1. mkdir -p /watson/archives

4. Copy the installation tarball onto the server into.

5. Untar the tarball that contains installation files.mkdir /opt/IBM/dataexplorer/WATSON_EXPR_ADV_EAC_V11.0.2_LNX_M
   cd /opt/IBM/dataexplorer /WATSON_EXPR_ADV_EAC_V11.0.2_LNX_M
   tar -xvf ./watson/archives/WATSON_EXPR_ADV_EAC_V11.0.2_LNX_M.tar

WEX AE Installation Details:

1. (Note must be performed with root level access) Install the software. (Note: Responses for Master node when selecting server type)

1. 1. cd /opt/IBM/dataexplorer/WATSON_EXPR_ADV_EAC_V11.0.2_LNX_M/
   2. ./install.bin -i console

• Installation Responses: (Also see installation screenshots below)
  • Choose Locate : English
  • PRESS TO CONTINUE:
  • Press Enter to continue view license agreement or 1 to accept: 1 (accept)
  • Enter fully qualified hostname of this server: (Hostname will be detected, hit enter. see example below)

• • Administrator user name: (Default: esadmin): (Hit enter to take default)

• • Create user account Enter the Number for your choice..: 2 ( the administrator user account ‘esadmin’ already exists)

• • Enter Administrator Account password: your_esadmin_password that you created for the user earlier (esadmin12)

• • Select a Server Type: (Response depends on type of server being installed on)
    • (1 –Master all on one server )

If installation is consists of a single server.

• • • (2 -Master Distributed Server )

If installation consists of multiple servers, and the current server is the Master server

• • • (3 -Additional Server )

If the installation is for a server (ie.. Search Server) that will connect to a Master

Select: 1

• • Do you want to install advanced options: 1 (YES)
  • Enter absolute path of the data directory ES_NODE_ROOT (default: /opt/IBM/dataexplorer/esdata ) /opt/IBM/dataexplorer/esdata
  • Enter the absolute path to install : /opt/IBM/dataexplorer/es

• • Common Communication Layer Port (default: 6002) Hit enter for the default

• • Search Server ESSearchServer Port (default: 8394) Hit enter for the default

• • Web Application Server (default embedded) 1 –embedded

• • Enterprise Search Application and Content Analytics Miner Port (default 8393) Hit enter for the default

• • THE INSTALLATION WILL BEGIN AND RUN ABOUT 5-10 MINUTES
    

…

• • This completes the WAC (Watson Content Analytic) installation
  • You can start esadmin to test:
    • /opt/IBM/es/bin/esadmin system startall
  • Go to admin console and check:
  • http://:8390/ESAdmin/

Install SIRE (Statistical Information and Relation Extraction) module

1. Stop esadmin – /opt/IBM/es/bin/esadmin system stopall

2. Iinstall the pre-req libraries:

yum -y install apr apr-util boost-filesystem boost-iostreams boost-program-options boost-regex boost-serialization

3. go to /opt/IBM/dataexplorer/es/bin/sire and run

rpm -ivh sire-20161109-1.x86_64.rpm

3. Stop esadmin – /opt/IBM/es/bin/esadmin system stopall
4. Start esadmin – /opt/IBM/es/bin/esadmin system startall
5. Installation is complete

Uninstall Notes

To Uninstall the WAC application (Note must be performed with root access), switch to root level account and perform the command:

/opt/IBM/dataexplorer/es/uninstall_11.0.1.0./uninstall_11.0.1.0 -i console

WEX AE System Start-up / Shutdown

To stop the services:

1. /opt/IBM/es/bin/esadmin system stopall

To start the services:

1. opt/IBM/es/bin/startccl.sh -bg
2. /opt/IBM/es/bin/esadmin system start
3. /opt/IBM/es/bin/esadmin system startall

 

Implementing Natural Language Query with IBM Watson Explorer

If you have a Watson Explorer (WEX) collection and want to be able to handle with Natural Query Language, you need to know that since WEX release 11.0.1, it have a native component to handle with this – its the query-modifier service.

Basically, this service parse the queries and apply some strategies, transforming the query in Keywords that WEX can understand and apply in the queries. Lets suppose that user search is:

“I’m looking for a Java Developer that know Struts and Spring and work from Brazil.”

The service will extract the keywords, based on configurations, and will search for:

Java Developer + Struts + spring + Brazil

We need to keep in mind that NLQ is different from Cognitive. This service will not understand questions, it will just extract terms. If you are looking for cognitive, you are looking for Watson (https://www.ibm.com/watson/developercloud/). With Watson we can understand the text and apply filter using location, range, etc. This also can be done using Machine Learning Models created at Watson Knowledge studio. But, Ill talk about this soon.

Backing to Query-Modifier, if you look at the folder nlq, inside Engine folder from your WEX installation, you will find the configuration stuff. Query Modifier work this way:

You make a request to WEX telling that you will use QM, the request pass through QM that apply the strategies, then, it forward the request to WEX Engine, who respond to you.

Here is a simple REST call that is using query-modifier:

http://MY_SERVER:9080/vivisimo/cgi-bin/velocity?v.app=api-rest&v.username=MY_USER&v.password=MY_PASSWORD&v.indent=true&v.function=query-search&fetch-timeout=30000&output-display-mode=limited&arena=MY_ARENA&output-contents-mode=list&syntax-operators=and+or+%28%29+CONTAINING+CONTENT+%25field%25%3A+%2B+NEAR+-+NOT+NOTCONTAINING+NOTWITHIN+OR0+quotes+regex+stem+THRU+BEFORE+FOLLOWEDBY+weight+wildcard+wildchar+WITHIN+WORDS+site+less-than+less-than-or-equal+greater-than+greater-than-or-equal+equal+range&sources=MY_COLLECTION+&output-contents=FIELD1+FIELD2&output-bold-contents=FIELD1&query=java+developer&query-condition-xpath=%24CONDITION_EXAMPLE=%27true%27&query-object=&num-per-source=20&start=0&num=20&query-modification-macros=enhance-query-with-querymodifier

See that the following make WEX use Query Modifier:

&query-modification-macros=enhance-query-with-querymodifier

In order to configure, go to <your WEX install folder>/Engine/nlq , in my case /opt/IBM/dataexplorer/WEX-11_0_1/Engine/nlq

Run “chmod +x querymodifier-install.sh”

Then “./querymodifier-install.sh” (as root)

You will see this kind of output:

Copying /opt/IBM/dataexplorer/WEX-11_0_1/Engine/examples/nlq/querymodifier/querymodifier-production.yml.defaults to /opt/IBM/dataexplorer/WEX-11_0_1/Engine/nlq/querymodifier-production.yml…

Configuring port to 9080…

Configuring path to vivisimo/cgi-bin/velocity…

Configuring PEARs path to /opt/IBM/dataexplorer/WEX-11_0_1/Engine/data/pears…

Copying querymodifier-2.1.9.jar to /opt/IBM/dataexplorer/WEX-11_0_1/Engine/nlq/querymodifier.jar…

Giving executable permissions to /opt/IBM/dataexplorer/WEX-11_0_1/Engine/nlq/querymodifier.jar…

Removing any existing /etc/init.d/querymodifier…

Linking /etc/init.d/querymodifier to …

Done.

Its important to change owner of the created files to WEX instance owner, in my case dataexp, so, as root: chown -R dataexp: <your WEX install folder>/Engine/nlq/

The configuration file is called querymodifier-production.yaml

In the first part of the file, you will see the WEX server setting, like IP, port and user.

After this you can setup the strategies, in my case I have this setup:

#The strategies to apply, by default, to each query. Can also be customized on a per-request basis (“workplan” GET parameter):

strategies:

default: PhraseWhitelistStrategy POSBasedNoiseWordRemoverStrategy DictionaryBasedNoiseWordRemoverStrategy DisjunctifyStrategy

The first strategy it the Disjunctify. It converts AND operators into OR operators, if the operator has more terms than a threshold. For example, if you set minimumRequiredTerms = 4, if user search for less terms than 4, query will be (A AND B AND C AND D), if you search for more than 4 terms, query will be (A OR B OR C OR D OR X OR …..).

The Dictionary-Based Noiseword Removal strategy, basically remove words from the query. For example, if you add BANANA to the list, then if user search for BANANA, it will be ignored. Usually we add to this section the common STOPWORDS, you can find several lists, I recommend use the google one. Another good list is here.

The Phrase Whitelist Strategy its interesting, you can have some external config files for some keyphrases, for example, lets suppose that you want that “Project Manager” be searched and “Project Manager”, and not “Project” and “Manager”, so, you need to add this word in the config file.

We have a secret here: you need to separate the words with <TAB> instead of space, else it will not work.

After configure your strategies, you just need to start the service (usually /etc/init.d/query-modifier start) and perform the REST Calls to test. You can follow the log at /var/log/querymodifier.log.

Every time that you change this setting, you need to recycle query modifier.

Your best friend to help with development and test, its the Api Runner interface from WEX engine. You can access this at:

http://YOUR_SERVER:9080/vivisimo/cgi-bin/velocity?v.app=api-run&v.function=query-parse-querymodifier

See the parameters there and ENJOY!

For more references: http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.1/com.ibm.watson.wex.fc.nlq.doc/c_wex_adding_nlq.html

Documentação de Perl

Estou tendo que aprender Perl para meu projeto na IBM, encontrei O Guia!

Eis o link: http://perldoc.perl.org/perlintro.html

Enjoy!

Introdução ao DB2

Para quem quiser conhecer mais este banco de dados, seguem um link sensacional:

http://www.ibm.com/developerworks/data/library/techarticle/0301chong/0301chong.html

Core J2EE Patterns

Interessado em Design Patterns? Encontrei a mina:

http://java.sun.com/blueprints/corej2eepatterns/Patterns/

Enjoy!

Guia de programação para RAD 7.5

Para quem utiliza as ferramentas da Rational (Rational Application Developer – RAD e Rational Software Architect – RSA) para desenvolvimento, a IBM acabou de disponibilizar gratuitamente um livro de programação para o RAD 7.5.

O link para download é: http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247672.html?Open

Vale a pena quem trabalha com o Eclipse dar uma olhada, uma vez que as ferramentas Rational são baseadas no mesmo.

Se você quiser levar estas ferramentas para sua faculdade, gratuitamente, veja como em http://www.ibm.com/br/university/

Enjoy

Manual do Ubuntu 8.04

A atualização do Guia Ubuntu ja foi feita para contemplar a versão 8.04. Este guia é muito interessante pois sua abordagem é bem prática.

Recomendado para quem quer iniciar ou aprimorar os conhecimentos no mundo do Ubuntu! Para ver o guia, clique no seguinte link: http://ubuntuguide.org/wiki/Ubuntu:Hardy

Aproveitando o post, eu percebi uma melhora na velocidade e correções de pequenos bugs que me irritavam nesse release. Só fiquei chateado com algumas configurações que eu havia feito (para o X por exemplo), que foram sobrescritas e tive que fazer novamente.

De resto, recomendo o upgrade!

Enjoy.

WebSphere Information Center: Tudo o que você precisa!

Assim como fiz com o DB2, estou postando o link para o Information Center do WebSphere Application Server.

O Information Center é o local no qual você pode encontrar tudo o que você precisa para trabalhar com o produto. É uma espécie de manual completo, com tutoriais, exemplos, etc.

Link para a versão 6.0 e Link para a versão 6.1.

Muita gente ainda utiliza as versões 5.0 ou 5.1, para elas, o infocenter continua online! 🙂

Enjoy.

Manual de CVS

Eu utilizo CVS diariamente em meu projeto atual, e não é simplementes para controlar alterações simuntâneas como muita gente faz. Nós utilizamos tags e branches. Pretendo escrever algo sobre isso em breve, pois certamente, muita gente até desconhece estas funcionalidades.

Neste site: http://ximbiot.com/cvs/wiki , eu encontrei muito material sobre CVS, com muitos exemplos. É um ótimo bookmark para desenvolvedores!

No meu post anterior falo de um cliente que pode ser utilizado em Linux/WIN/MAC, muito bom mesmo! Recomendo!

Bom trabalho!

Manual completo do Ubuntu

Encontrei um manual completo sobre o Ubuntu aqui: http://ubuntuguide.org/wiki/Ubuntu:Gutsy

Ele foi iniciado por Chua Wen Kiat (Kuala Lumpur, Malásia) e está sendo mantido pela Universidade da Letonia mas todos estão convidados a contribuir.

Tem praticamente TUDO que um usuário do Ubuntu Linux precisa! O legal é que a versão em português está evoluindo bem, logo estará completa

Performance e Tunning

Aproveitando o post, aqui vai um link muito bom que mostra como melhorar a performance de seu ubuntu: http://www.extremetech.com/article2/0,1697,2114123,00.asp

Barra estilo MAC OS X

Para quem gosta da barra estilo do MAC, Edivaldo Brito escreveu um post show aqui: http://www.edivaldobrito.com.br/?p=54

A sua barra vai ficar como a abaixo:

É interessante adicionar o comando cairo-dock na inicialização do seu gnome, para fazer isso, execute no terminal “gnome-session–startup” e adicione o cairo-dock, desta forma, ao abrir o gnome, a barra é carregada automaticamente!