Ruby Blues in Openshift Origin v2 (or How I broke my app by installing new relic gem)

The day I tried to install a Ruby gem for New Relic RPM, it causes almost a day  downtime for Phusion Passenger-run Rails Application. It turns out that there are several thing that worth noting about Ruby dependency management.

The First Error

The error message that causes downtime is like this :

Ruby (Rack) application could not be started

These are the possible causes:
  • There may be a syntax error in the application's code. Please check for such errors and fix them.
  • A required library may not installed. Please install all libraries that this application requires.
  • The application may not be properly configured. Please check whether all configuration files are written correctly, fix any incorrect configurations, and restart this application.
  • A service that the application relies on (such as the database server or the Ferret search engine server) may not have been started. Please start that service.
Further information about the error may have been written to the application's log file. Please check it in order to analyse the problem.
Error message:
Could not find newrelic_rpm-3.18.1.330 in any of the sources (Bundler::GemNotFound)
Exception class:
PhusionPassenger::UnknownError
Application root:
/var/www/openshift/console
The nature of this error is tightly coupled to how 'bundle' and 'gem install' works.

Bundle install and Gem.lock

If we execute bundle install, bundle would :
1) connect to the internet to check for latest gem versions
2) Sometimes update the gem.lock (need to explore further when it does) when finding newer gem version
3) install the gem using standard gem environment
These 3 might break your application during some circumstances. In my case, the environment running the application is a bit different than the environment in the command line.

Gem environment

The result of running 'gem environment' tells much about the situation :

RubyGems Environment:
  - RUBYGEMS VERSION: 1.8.24
  - RUBY VERSION: 1.9.3 (2013-11-22 patchlevel 484) [x86_64-linux]
  - INSTALLATION DIRECTORY: /opt/rh/ruby193/root/usr/local/share/gems
  - RUBY EXECUTABLE: /opt/rh/ruby193/root/usr/bin/ruby
  - EXECUTABLE DIRECTORY: /opt/rh/ruby193/root/usr/local/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /opt/rh/ruby193/root/usr/local/share/gems
     - /root/.gem/ruby/1.9.1
     - /opt/rh/ruby193/root/usr/share/gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - http://rubygems.org/

Things to note here, the INSTALLATION DIRECTORY and GEM PATHS difference. The /usr/local/share/gems exist in GEM PATHS, as well as /usr/share/gems, but INSTALLATION will put the gems under /usr/local/share/gems.

The environment for running Ruby is a bit different too :


broker ~ # cat /var/www/openshift/console/script/console_ruby
export LD_LIBRARY_PATH=/opt/rh/ruby193/root/usr/local/lib64:/opt/rh/ruby193/root/usr/lib64:/opt/rh/v8314/root/usr/lib64
export GEM_HOME=/opt/rh/ruby193/root/usr/share/gems
export GEM_PATH=/opt/rh/root/usr/local/share/gems:/opt/rh/ruby193/root/usr/share/gems

ruby193-ruby $@

So the first GEM_PATH missed ruby193 prefix. Without the ruby193 prefix, the gems inside the actual directory would not be found, causing the error. Whats with the local directory anyway, this is not the first time I get confused by Linux/Unix directory scheme. It has historical significance but IMHO it is better not to have too many directories nowadays.

Solution to Problem I (or how to install newrelic_rpm in openshift broker&console)

Instead of fixing the GEM_PATH, I opt to use the install gem in the /usr/share/gems directory by using --install-dir :


gem install newrelic_rpm -v 3.18.1.330 --install-dir=/opt/rh/ruby193/root/usr/share/gems

Then add this one-liner in /var/www/openshift/broker/Gemfile

gem 'newrelic_rpm', '3.18.1.330'

Second error

Actually, before finding the solution above, another error crops up with these symptoms :


broker console # bundle install --local
Could not find rake-0.9.2.2 in any of the sources
broker console # bundle install
Fetching gem metadata from https://rubygems.org/...........
Fetching gem metadata from https://rubygems.org/..
Could not find openshift-origin-console-1.26.3.1 in any of the sources

As someone who are not familiar Ruby development, I found these two errors confusing. The first error, where bundle cannot find rake package, is very strange, because the 0.9.2.2 package is there :
broker console # gem list rake

*** LOCAL GEMS ***

rake (0.9.2.2)
broker console # ls -l /opt/rh/ruby193/root/usr/share/gems/gems | grep rake
drwxr-xr-x.  4 root root 4096 Nov 18  2014 rake-0.9.2.2
broker console #

Because version 0.9.2.2 is in available in the 'net (see https://rubygems.org/gems/rake/versions), I tried to use the internet by removing the --local keyword, and this time bundle complains about openshift-origin-console, which is not avaiable in the great internet.

The clue for the second error is the result of bundle config command :
broker console # bundle config
Settings are listed in order of priority. The top value will be used.

path
Set for your local app (/var/www/openshift/console/.bundle/config): "vendor"

disable_shared_gems
Set for your local app (/var/www/openshift/console/.bundle/config): "1"

Which is not the same with the neighboring application which has no settings at all:

broker broker # bundle config
Settings are listed in order of priority. The top value will be used.

broker broker # 

Solution to problem II

The solution is to remove .bundle/config file, which inadvertently created when I tried running bundle install --path vendor. The file redirects local gem searches into the ./vendor directory, thus skipping /opt/rh/ruby193/root/usr/share/gems directory and causing the 2nd error. The bundle program are unable to find the openshift gem location because it were searching the ./vendor directory.


broker console # ls -al
total 104
drwxr-x---. 14 apache apache  4096 Dec 10 20:46 .
drwxr-xr-x.  4 root   root    4096 Jun 15  2014 ..
drwxr-x---.  8 apache apache  4096 Dec  8 08:05 app
drwxr-xr-x.  2 root   root    4096 Dec 10 20:46 .bundle
drwxr-x---.  5 apache apache  4096 Dec  8 08:38 config
-rw-r-----.  1 apache apache   166 Jul 11  2014 config.ru
-rw-r-----.  1 apache apache   809 Dec  8 08:37 Gemfile
-rw-r--r--.  1 root   root    3487 Dec 10 20:31 Gemfile.lock
-rw-r--r--.  1 root   root    3453 Dec  8 07:18 Gemfile.lock.copy
drwxr-x---.  6 apache apache  4096 Dec  9 13:12 httpd
drwxr-x---.  2 apache apache  4096 Dec  8 08:38 log
drwxr-x---.  2 apache apache  4096 Jul 11  2014 .openshift
-rw-r-----.  1 apache apache 11754 Jul 11  2014 openshift-origin-console.spec
drwxr-x---.  3 apache apache  4096 Dec  8 08:05 public
-rw-r-----.  1 apache apache   398 Jul 11  2014 Rakefile
-rw-r-----.  1 apache apache  9208 Jul 11  2014 README.rdoc
drwxr-x---.  2 apache apache  4096 Jul 11  2014 run
drwxr-x---.  2 apache apache  4096 Dec  9 13:12 script
drwxr-x---.  7 apache apache  4096 Dec  8 08:05 test
drwxr-x---.  6 apache apache  4096 Dec  8 08:05 tmp
drwxr-x---.  6 apache apache  4096 Dec 10 20:46 vendor
-rw-r--r--.  1 root   root    1485 Dec 10 20:27 versionlist.log
broker console # rm -rf .bundle
broker console #

Now I can do bundle install --local again..

Lessons learned

What I learned, that is : bundle install are evil. It  install new gems and change application dependencies, sometimes this breaks your app (especially if your app is written by Red Hat and has missed some GEM_PATH). You must understood bundle's basic mechanism, because incorrect parameters will lead to breaking your application.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more