Migrating Joget 3 Instance

On some occassions we need to move joget 3 workflow service from one server to another. We need to migrate two part : the server application files, and the database. Our aim is to migrate joget server with data intact.

Identify running instance

First we need to determine where joget is installed. In an ideal situation this information is stored in CMDB. But since we have yet to see Joget - Cloud foundry package, and we haven't deploy CMDB enterprisewide, this step is necessary.
Look  for java processes in the source machine :
ps auxw | grep java
In my machine we got these output :
501      26733  0.0 22.2 759176 457552 ?       Sl   Apr26  18:20 /home/admin-dev/p/jdk1.6.0_21/bin/java -XX:MaxPermSize=128m -Xmx512M -Dwflow.home=./wflow/ -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/conf/logging.properties -Djava.endorsed.dirs=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/endorsed -classpath :/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/bin/bootstrap.jar -Dcatalina.base=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18 -Dcatalina.home=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18 -Djava.io.tmpdir=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/temp org.apache.catalina.startup.Bootstrap start

I really don't know why the user shown as 501 here. It should be the user name that the java process runs. Lets ignore that for now.
From the parameters we note that joget is installed in /home/admin-dev/p/joget-linux-3.0.0

Identify database connection

Lets check  where the Joget database is located :
[admin-dev@ead-dev wflow]$ cd /home/admin-dev/p/joget-linux-3.0.0/
[admin-dev@ead-dev joget-linux-3.0.0]$ cd wflow
[admin-dev@ead-dev wflow]$ ls
app_datasource-default.properties  app_datasource.properties  app_plugins
app_datasource-esshr.properties    app_forms                  app_xpdlImages
[admin-dev@ead-dev wflow]$

lets see the 'pointing' file :
[admin-dev@ead-dev wflow]$ cat app_datasource.properties
#
#Mon May 25 15:33:51 SGT 2009
currentProfile=esshr
 
 and the real database config is :
[admin-dev@ead-dev wflow]$ cat app_datasource-esshr.properties
workflowDriver=com.mysql.jdbc.Driver
workflowUrl=jdbc\:mysql\://10.65.10.150\:3306/jwdb?characterEncoding\=UTF-8
workflowUser=joget
profileName=
workflowPassword=
Now we're ready to start the migration.

Shutdown server


We want to minimize data inconsistency, so lets do a cold migration. Kill the joget server.

[admin-dev@ead-dev joget-linux-3.0.0]$ ./tomcat6.sh stop
Using CATALINA_BASE:   /home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18
Using CATALINA_HOME:   /home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18
Using CATALINA_TMPDIR: /home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/temp
Using JRE_HOME:       /home/admin-dev/p/jdk1.6.0_21
[admin-dev@ead-dev joget-linux-3.0.0]$
The shell script runs but says nothing about whether it succeed or not. Review the java process :
[admin-dev@ead-dev apache-tomcat-6.0.18]$ ps auxw | grep java
501      26238  0.0  0.0  61204   740 pts/1    R+   09:09   0:00 grep java
501      26733  0.0 22.4 758848 461972 ?       Sl   Apr26  18:30 /home/admin-dev/p/jdk1.6.0_21/bin/java -XX:MaxPermSize=128m -Xmx512M -Dwflow.home=./wflow/ -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/conf/logging.properties -Djava.endorsed.dirs=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/endorsed -classpath :/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/bin/bootstrap.jar -Dcatalina.base=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18 -Dcatalina.home=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18 -Djava.io.tmpdir=/home/admin-dev/p/joget-linux-3.0.0/apache-tomcat-6.0.18/temp org.apache.catalina.startup.Bootstrap start

Its still running. But I check the netstat also :
[admin-dev@ead-dev apache-tomcat-6.0.18]$ netstat -anp | grep 8080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
[admin-dev@ead-dev apache-tomcat-6.0.18]$

The 8080 port is closed. Its stopped. But the java process stuck somewhat. Lets fix that :
[admin-dev@ead-dev apache-tomcat-6.0.18]$ kill 26733
[admin-dev@ead-dev apache-tomcat-6.0.18]$ netstat -anp | grep 8080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
[admin-dev@ead-dev apache-tomcat-6.0.18]$ ps auxw | grep java
501      26252  0.0  0.0  61204   744 pts/1    R+   09:11   0:00 grep java

Copying joget server files


Now lets do a scp to target :

[admin-dev@ead-dev joget-linux-3.0.0]$ cd ..
[admin-dev@ead-dev p]$ pwd
/home/admin-dev/p
[admin-dev@ead-dev p]$ scp -r joget-linux-3.0.0/ admin-sppd@10.65.10.156:/home/admin-sppd/p
The authenticity of host '10.65.10.156 (10.65.10.156)' can't be established.
RSA key fingerprint is e2:a8:38:8a:5f:30:8a:77:24:57:b2:9a:9d:28:ef:6d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.65.10.156' (RSA) to the list of known hosts.
admin-sppd@10.65.10.156's password:
CHANGES.txt                                   100%   23KB  23.4KB/s   00:00
tmlog1310.log                                 100%  169KB 168.8KB/s   00:00
README.txt                                    100% 1716     1.7KB/s   00:00
tm.out                                        100%  251KB 250.7KB/s   00:00
...

 Database migration

Dump the database from one convenient server..

[admin-dev@ead-dev p]$ mysqldump -h 10.65.10.150 -u joget -p jwdb > jogetjwdb.sql
Enter password:
[admin-dev@ead-dev p]$ ls -l
total 86120
drwxrwxr-x 9 admin-dev admin-dev     4096 Feb 15 18:16 grinder-3.4
drwxr-xr-x 9 admin-dev admin-dev     4096 Feb 13 09:18 jdk1.6.0_21
-rw-rw-r-- 1 admin-dev admin-dev 26635581 May 31 09:17 jogetjwdb.sql
...



Create a new database in the target system (in this case, the same host as the original..)
-bash-3.2$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1421944
Server version: 5.0.77-log Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show databases;
...
mysql> create database jwdbprod;
Query OK, 1 row affected (0.29 sec)

mysql> grant all on jwdbprod.* to jogetprod@'%' identified by 'passwordhere';
Query OK, 0 rows affected (0.09 sec)

mysql>

And then we import the database from where we dumped it last :

[admin-dev@ead-dev p]$ mysql -u jogetprod -h 10.65.10.150 -p jwdbprod
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1422462
Server version: 5.0.77-log Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> source ./jogetjwdb.sql; 

Reconfigure joget

[admin-sppd@sppd-online1 ~]$ cd p/joget-linux-3.0.0/wflow/
[admin-sppd@sppd-online1 wflow]$ vi app_
app_datasource-default.properties  app_forms/
app_datasource-esshr.properties    app_plugins/
app_datasource.properties          app_xpdlImages/
[admin-sppd@sppd-online1 wflow]$ vi app_datasource-esshr.properties
[admin-sppd@sppd-online1 wflow]$

Ok.. lets start it up
[admin-sppd@sppd-online1 wflow]$ cd ..
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ ls
10.2.16.13.tm0.epoch  data                             tmlog1310.log
10.2.16.19.tm7.epoch  docs                             tm.out
127.0.0.1.tm30.epoch  d:\participantLog.joget.txt      tomcat6.sh
apache-ant-1.7.1      d:\participantLog.joget.txt.lck  VERSION.txt
apache-tomcat-6.0.18  LICENSE.txt                      wflow
build.xml             README.txt
CHANGES.txt           setup.sh
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ ./tomcat6.sh start
Using CATALINA_BASE:   /home/admin-sppd/p/joget-linux-3.0.0/apache-tomcat-6.0.18
Using CATALINA_HOME:   /home/admin-sppd/p/joget-linux-3.0.0/apache-tomcat-6.0.18
Using CATALINA_TMPDIR: /home/admin-sppd/p/joget-linux-3.0.0/apache-tomcat-6.0.18/temp
Using JRE_HOME:       /home/admin-dev/p/jdk1.6.0_21
No clue about startup process as usual..
Lets check the logs
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ cd apache-tomcat-6.0.18/
[admin-sppd@sppd-online1 apache-tomcat-6.0.18]$ cd logs
[admin-sppd@sppd-online1 logs]$ tail -f catalina.out
 the last lines were
INFO: Stopping Coyote HTTP/1.1 on http-8080
apache-tomcat-6.0.18/bin/catalina.sh: line 292: /home/admin-dev/p/jdk1.6.0_21/bin/java: No such file or directory

There is some hardwired path that we must fix (this might not be exist in your installation, in my installation I changed some shell scripts). Check where our jdk is :

[admin-sppd@sppd-online1 joget-linux-3.0.0]$ which java
~/p/jdk1.6.0_31/bin/java
 Now reconfig  the path (note different home dir and jdk version)
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ vi tomcat6.sh
#/bin/sh

export JAVA_OPTS="-XX:MaxPermSize=128m -Xmx512M -Dwflow.home=./wflow/ "
#export JAVA_HOME=/usr/java/jdk1.6.0_21
export JAVA_HOME=/home/admin-sppd/p/jdk1.6.0_31
apache-tomcat-6.0.18/bin/catalina.sh $*

And try start again. but another problem looms :

[admin-sppd@sppd-online1 joget-linux-3.0.0]$ tail -n 100 -f apache-tomcat-6.0.18/logs/catalina.out
May 31, 2012 9:41:26 AM org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use:8080
        at org.apache.catalina.connector.Connector.start(Connector.java:1138)
        at org.apache.catalina.core.StandardService.start(StandardService.java:531)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

The port 8080 is in use.
What app is using it ?
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ netstat -anp | grep 8080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      28482/java
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ ps auxw | grep 28482
500       7345  0.0  0.0  61204   744 pts/2    R+   09:43   0:00 grep 28482
500      28482  0.0 10.4 920632 215888 ?       Sl   Feb17  42:03 /home/admin-sppd/p/jdk1.6.0_31/bin/java -Djava.util.logging.config.file=/home/admin-sppd/p/vfabric-tc-server-developer/insight-instance/conf/logging.properties -Xmx512M -Xss192K -Dinsight.base=/home/admin-sppd/p/vfabric-tc-server-developer/insight-instance/insight -Dinsight.logs=/home/admin-sppd/p/vfabric-tc-server-developer/insight-instance/logs -Djava.awt.headless=true -Dgemfire.disableShutdownHook=true -Dgemfire.ignoreMisses=true -XX:MaxPermSize=256m -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Djava.endorsed.dirs=/home/admin-sppd/p/vfabric-tc-server-developer/tomcat-7.0.23.A.RELEASE/endorsed -classpath /home/admin-sppd/p/vfabric-tc-server-developer/insight-instance/bin/insight-bootstrap-tcserver-1.5.1.SR2.jar:/home/admin-sppd/p/vfabric-tc-server-developer/insight-instance/lib/aspectjweaver-1.6.11.SR1.jar:/home/admin-sppd/p/vfabric-tc-server-developer/tomcat-7.0.23.A.RELEASE/bin/bootstrap.jar:/home/admin-sppd/p/vfabric-tc-server-developer/tomcat-7.0.23.A.RELEASE/bin/tomcat-juli.jar -Dcatalina.base=/home/admin-sppd/p/vfabric-tc-server-developer/insight-instance -Dcatalina.home=/home/admin-sppd/p/vfabric-tc-server-developer/tomcat-7.0.23.A.RELEASE -Djava.io.tmpdir=/home/admin-sppd/p/vfabric-tc-server-developer/insight-instance/temp org.apache.catalina.startup.Bootstrap start

Ah. A vfabric tc server developer. I don't think we have any need of it running in the production server. Lets kill it.
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ kill 28482

and restart the Joget's tomcat

[admin-sppd@sppd-online1 joget-linux-3.0.0]$ ./tomcat6.sh stop                  [admin-sppd@sppd-online1 joget-linux-3.0.0]$ ps auxw | grep java
500       7388  0.0  0.0  61204   744 pts/2    R+   09:47   0:00 grep java
[admin-sppd@sppd-online1 joget-linux-3.0.0]$
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ ./tomcat6.sh start
Using CATALINA_BASE:   /home/admin-sppd/p/joget-linux-3.0.0/apache-tomcat-6.0.18
Using CATALINA_HOME:   /home/admin-sppd/p/joget-linux-3.0.0/apache-tomcat-6.0.18
Using CATALINA_TMPDIR: /home/admin-sppd/p/joget-linux-3.0.0/apache-tomcat-6.0.18/temp
Using JRE_HOME:       /home/admin-sppd/p/jdk1.6.0_31

Check the logs.. 
oh no, still another problem :

[admin-sppd@sppd-online1 joget-linux-3.0.0]$ tail -n 100 -f apache-tomcat-6.0.18/logs/catalina.out 
ERROR 31 May 2012 09:49:32 org.hibernate.util.JDBCExceptionReporter  - Cannot create PoolableConnectionFactory (Access denied for user 'jogetprod'@'%' to database 'jwdb')
ERROR 31 May 2012 09:49:32 org.springframework.web.context.ContextLoader  - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'setupSessionFactory' defined in class path resource [commonsApplicationContext.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Hibernate Dialect must be explicitly set
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1338)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
Seems I overlook database name in workflowUrl parameter. Editing that line in   wflow/app_datasource-esshr.properties, and restart again with tomcat6.sh stop and start. The result :
[admin-sppd@sppd-online1 joget-linux-3.0.0]$ tail -f apache-tomcat-6.0.18/logs/catalina.out
May 31, 2012 9:58:07 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive spring2.war
May 31, 2012 9:58:10 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
May 31, 2012 9:58:10 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
May 31, 2012 9:58:10 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/15  config=null
May 31, 2012 9:58:10 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 25291 ms
Ok. lets see the joget web.


Seems that the migration is done.
Thats all for now.. I hope your migration succeed as well.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more