In an attempt to get an older version of Chef (12.22.5) working on a newer operating system (Ubuntu 20.04) I came across a segmentation fault caused by an OpenSSL version mismatch between the embedded version in Chef Client and the system version of OpenSSL in Ubuntu 20.04. This would typically be a relatively easy fix by upgrading to Chef 13+ in order to properly support the newer Ubuntu. However, because of various reasons, this was not an option.
The issue was originally opened in Chef's Github (#6485) back in 2017 and instead of upgrading Chef, we decided to attempt to rebuild Chef Client with a newer version of OpenSSL (as mentioned in this comment). However, I couldn't find much information on how to actually do it. Here we'll be building a custom Chef 12.22.5 deb package which includes OpenSSL 1.1.1f and Ruby 2.4.0. Installation of this package was tested and appears to resolve the issue.
Disclaimer
I recommend creating a new Ubuntu 20.04 virtual machine to complete the steps below. You should have the expectation you'll need to delete it and try again. This process involves installing and building relatively old software that almost certainly isn't supported anymore.
Resources
Chef v12.22.5 Github: https://github.com/chef/chef/tree/v12.22.5
Chef v12.22.5 Omnibus README: https://github.com/chef/chef/blob/v12.22.5/omnibus/README.md
Overview
Chef recommends we run `rake dependencies` to update any packages or Gems when building chef-client, however, I wasn't able to get this to work and reverted to the more "manual" method below. If you're able to get `rake dependencies` to complete, it may make this process a little simpler.
As mentioned above, the primary issue is with the embedded version of OpenSSL in Chef Client 12.22.5. Postgres, and ultimately PG gem's usage of Postgres, uses the newer OpenSSL (1.1+) provided with the operating system (Ubuntu 20.04 in this case). The problem arises because there's essentially a mismatch of OpenSSL libraries between Chef and Postgres, causing a segfault like below...
* postgresql_database[artifactory] action createThe PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.
You should use PG::Connection, PG::Result, and PG::Error instead, respectively.
Called from /var/chef/cache/cookbooks/database/libraries/provider_database_postgresql.rb:132:in `db'
/var/chef/cache/cookbooks/database/libraries/provider_database_postgresql.rb:132: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0043 p:---- s:0216 e:000215 CFUNC :initialize
c:0042 p:---- s:0213 e:000212 CFUNC :new
c:0041 p:0203 s:0208 e:000207 METHOD /var/chef/cache/cookbooks/database/libraries/provider_database_postgresql.rb:132
c:0040 p:0044 s:0199 e:000198 METHOD /var/chef/cache/cookbooks/database/libraries/provider_database_postgresql.rb:97
c:0039 p:0008 s:0194 e:000193 LAMBDA /var/chef/cache/cookbooks/database/libraries/provider_database_postgresql.rb:46 [FINISH]
...
-- C level backtrace information -------------------------------------------
/opt/chef/embedded/lib/libruby.so.2.4(rb_vm_bugreport+0x559) [0x7f4067ed6e59] vm_dump.c:684
/opt/chef/embedded/lib/libruby.so.2.4(rb_bug_context+0xd0) [0x7f4067d6e1d0] error.c:506
/opt/chef/embedded/lib/libruby.so.2.4(sigsegv+0x3e) [0x7f4067e4f93e] signal.c:907
/lib/x86_64-linux-gnu/libpthread.so.0 [0x7f4067ac50c0]
/lib/x86_64-linux-gnu/libc.so.6 [0x7f4066ef6386]
/opt/chef/embedded/lib/libcrypto.so.1.0.0(getrn+0x89) [0x7f40652c7339]
/opt/chef/embedded/lib/libcrypto.so.1.0.0(lh_insert+0x42) [0x7f40652c7602]
/opt/chef/embedded/lib/libcrypto.so.1.0.0(OBJ_NAME_add+0x85) [0x7f4065223625]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 [0x7f405b3dca3c]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 [0x7f405b3f40d9]
/lib/x86_64-linux-gnu/libpthread.so.0 [0x7f4067ac2739]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0x9) [0x7f405b449f59]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(OPENSSL_init_crypto+0x1b3) [0x7f405b3f4533]
/usr/lib/x86_64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x74) [0x7f405b74bbd4]
/usr/lib/x86_64-linux-gnu/libpq.so.5 [0x7f405b9b1a8c]
/usr/lib/x86_64-linux-gnu/libpq.so.5(PQconnectPoll+0xbbe) [0x7f405b999abe]
/usr/lib/x86_64-linux-gnu/libpq.so.5 [0x7f405b99a57a]
/usr/lib/x86_64-linux-gnu/libpq.so.5(PQconnectdb+0x27) [0x7f405b99d867]
/opt/chef/embedded/lib/ruby/gems/2.4.0/gems/pg-0.21.0/lib/pg_ext.so(gvl_PQconnectdb_skeleton+0xc) [0x7f405bbdc70c] gvl_wrappers.c:9
/opt/chef/embedded/lib/libruby.so.2.4(rb_thread_call_without_gvl+0x3b) [0x7f4067e8f5db] thread.c:1320
/opt/chef/embedded/lib/ruby/gems/2.4.0/gems/pg-0.21.0/lib/pg_ext.so(gvl_PQconnectdb+0x29) [0x7f405bbdcb19] gvl_wrappers.c:10
/opt/chef/embedded/lib/ruby/gems/2.4.0/gems/pg-0.21.0/lib/pg_ext.so(pgconn_init+0x5a) [0x7f405bbe513a] pg_connection.c:283We can reproduce the issue fairly quickly with the below code...
#!/opt/chef/embedded/bin/ruby
require 'openssl'
require 'pg'
::PGconn.new( host: 'localhost', port: 22 )Downgrading the OpenSSL version used in Ubuntu 20.04 will not work because many other packages rely on OpenSSL 1.1+ and are incompatible with older versions. Installing both versions isn't recommended either.
The Fix
We decided to rebuild the chef-client package to include a newer version of OpenSSL. Upgrading the Chef 13 wasn't an option at that point so the below was attempted. This is admittedly a bit of a hacky solution but I found it to be a really interesting challenge.
1) Install Ruby 2.4.0 via RVM
# Install & configure RVM
sudo gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
sudo apt-get install software-properties-common
sudo apt-add-repository -y ppa:rael-gc/rvm
sudo apt-get update
sudo apt-get install -y rvm
echo 'source "/etc/profile.d/rvm.sh"' >> ~/.bashrc
source ~/.bashrc
# Make sure permissions are set correctly for your user
rvmsudo rvm get stable --auto-dotfiles
rvm reload
rvm fix-permissions system
rvm group add rvm $USER
logout
# Log back in and install Ruby 2.4.0
rvm install 2.4.0
rvm use 2.4.0
# Make sure the bundler version is 1.17.3
bundle --version2) Clone Chef and checkout the latest 12.22.5 branch
git clone https://github.com/chef/chef.git ~/chef
cd ~/chef && git checkout 53c4a6cd48045d1de88a688cf719bb6760d3021f3) Make sure you have the following bundler configuration in ~/chef/.bundle/config
mkdir -p ~/chef/.bundle/config
echo '---
BUNDLE_FROZEN: "true"
BUNDLE_WITHOUT: "developmentvi:omnibus_overrides.rb:omnibus_package:test:pry:integration:docgen:maintenance:travis:aix:bsd:linux:mac_os_x:solaris:windows:development"' >> ~/chef/.bundle/config4) Install the necessary gems for Chef and Omnibus. Make sure all Gems are installed successfully.
cd ~/chef && bundle install
cd ~/chef/omnibus && bundle install --without development5) Update the OpenSSL and Ruby versions in the relevant configuration files
--- a/version_policy.rb
+++ b/version_policy.rb
@@ -33,7 +33,7 @@ OMNIBUS_OVERRIDES = {
"makedepend" => "1.0.5",
"ncurses" => "5.9",
"pkg-config-lite" => "0.28-1",
- "ruby" => "2.3.6",
+ "ruby" => "2.4.0",
# Leave dev-kit pinned to 4.5 on 32-bit, because 4.7 is 20MB larger and we don't want
# to unnecessarily make the client any fatter. (Since it's different between
# 32 and 64, we have to do it in the project file still.)
@@ -45,7 +45,7 @@ OMNIBUS_OVERRIDES = {
## These can float as they are frequently updated in a way that works for us
#override "cacerts" =>"???",
- "openssl" => "1.0.2n",
+ "openssl" => "1.1.1f",
--- a/omnibus_overrides.rb
+++ b/omnibus_overrides.rb
@@ -11,9 +11,9 @@ override "libyaml", version: "0.1.7"
override "makedepend", version: "1.0.5"
override "ncurses", version: "5.9"
override "pkg-config-lite", version: "0.28-1"
-override "ruby", version: "2.3.6"
+override "ruby", version: "2.4.0"
override "ruby-windows-devkit-bash", version: "3.1.23-4-msys-1.0.18"
override "util-macros", version: "1.19.0"
override "xproto", version: "7.0.28"
override "zlib", version: "1.2.11"
-override "openssl", version: "1.0.2n"
+override "openssl", version: "1.1.1f"
6) Disable caching in the Omnibus config (~/chef/omnibus/omnibus.rb), this forces the download of OpenSSL from the source defined in our custom version-manifest.json (step 7)
--- a/omnibus/omnibus.rb
+++ b/omnibus/omnibus.rb
@@ -37,7 +37,7 @@ windows_arch env_omnibus_windows_arch
# Enable S3 asset caching
# ------------------------------
-use_s3_caching true
+use_s3_caching false
s3_access_key ENV["AWS_ACCESS_KEY_ID"]
s3_secret_key ENV["AWS_SECRET_ACCESS_KEY"]
s3_bucket "opscode-omnibus-cache"7) Create our custom version-manifest.json in ~/chef/omnibus/pkg This contains the version locks for Ruby and OpenSSL and the SHA256 hash for the downloaded OpenSSL package.
mkdir -p ~/chef/omnibus/pkg
echo '{"manifest_format":2,"software":{"preparation":{"locked_version":"1.0.0","locked_source":null,"source_type":"project_local","described_version":"1.0.0","license":"project_license"},"zlib":{"locked_version":"1.2.11","locked_source":{"md5":"1c9f62f0778697a09d36121ead88e08e","url":"http://downloads.sourceforge.net/project/libpng/zlib/1.2.11/zlib-1.2.11.tar.gz"},"source_type":"url","described_version":"1.2.11","license":"Zlib"},"liblzma":{"locked_version":"5.2.3","locked_source":{"md5":"ef68674fb47a8b8e741b34e429d86e9d","url":"http://tukaani.org/xz/xz-5.2.3.tar.gz"},"source_type":"url","described_version":"5.2.3","license":"Public-Domain"},"config_guess":{"locked_version":"84f04b02a7e2fc8eaa9d52deee5f6d57b06fe447","locked_source":{"git":"https://github.com/chef/config-mirror.git"},"source_type":"git","described_version":"master","license":"GPL-3.0 (with exception)"},"libxml2":{"locked_version":"2.9.7","locked_source":{"sha256":"f63c5e7d30362ed28b38bfa1ac6313f9a80230720b7fb6c80575eeab3ff5900c","url":"ftp://xmlsoft.org/libxml2/libxml2-2.9.7.tar.gz"},"source_type":"url","described_version":"2.9.7","license":"MIT"},"libxslt":{"locked_version":"1.1.30","locked_source":{"sha256":"ba65236116de8326d83378b2bd929879fa185195bc530b9d1aba72107910b6b3","url":"ftp://xmlsoft.org/libxml2/libxslt-1.1.30.tar.gz"},"source_type":"url","described_version":"1.1.30","license":"MIT"},"libiconv":{"locked_version":"1.15","locked_source":{"url":"https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.15.tar.gz","sha256":"ccf536620a45458d26ba83887a983b96827001e92a13847b45e4925cc8913178"},"source_type":"url","described_version":"1.15","license":"LGPL-2.1"},"cacerts":{"locked_version":"2018-01-17","locked_source":{"url":"https://curl.haxx.se/ca/cacert-2018-01-17.pem","sha256":"defe310a0184a12e4b1b3d147f1d77395dd7a09e3428373d019bef5d542ceba3"},"source_type":"url","described_version":"2018-01-17","license":"MPL-2.0"},"xproto":{"locked_version":"7.0.28","locked_source":{"md5":"0b42843b99aee3e4f6a9cc7710143f86","url":"https://www.x.org/releases/individual/proto/xproto-7.0.28.tar.gz"},"source_type":"url","described_version":"7.0.28","license":"MIT"},"util-macros":{"locked_version":"1.19.0","locked_source":{"md5":"40e1caa49a71a26e0aa68ddd00203717","url":"https://www.x.org/releases/individual/util/util-macros-1.19.0.tar.gz"},"source_type":"url","described_version":"1.19.0","license":"MIT"},"pkg-config-lite":{"locked_version":"0.28-1","locked_source":{"md5":"61f05feb6bab0a6bbfab4b6e3b2f44b6","url":"http://downloads.sourceforge.net/project/pkgconfiglite/0.28-1/pkg-config-lite-0.28-1.tar.gz"},"source_type":"url","described_version":"0.28-1","license":"GPL-2.0"},"makedepend":{"locked_version":"1.0.5","locked_source":{"url":"https://www.x.org/releases/individual/util/makedepend-1.0.5.tar.gz","md5":"efb2d7c7e22840947863efaedc175747"},"source_type":"url","described_version":"1.0.5","license":"MIT"},"openssl":{"locked_version":"1.1.1f","locked_source":{"url":"https://www.openssl.org/source/openssl-1.1.1f.tar.gz","extract":"lax_tar","sha256":"186c6bfe6ecfba7a5b48c47f8a1673d0f3b0e5ba2e25602dd23b629975da3f35"},"source_type":"url","described_version":"1.1.1f","license":"OpenSSL"},"libtool":{"locked_version":"2.4.2","locked_source":{"md5":"d2f3b7d4627e69e13514a40e72a24d50","url":"https://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz"},"source_type":"url","described_version":"2.4.2","license":"GPL-2.0"},"libffi":{"locked_version":"3.2.1","locked_source":{"md5":"83b89587607e3eb65c70d361f13bab43","url":"ftp://sourceware.org/pub/libffi/libffi-3.2.1.tar.gz"},"source_type":"url","described_version":"3.2.1","license":"MIT"},"libyaml":{"locked_version":"0.1.7","locked_source":{"sha256":"8088e457264a98ba451a90b8661fcb4f9d6f478f7265d48322a196cec2480729","url":"http://pyyaml.org/download/libyaml/yaml-0.1.7.tar.gz"},"source_type":"url","described_version":"0.1.7","license":"MIT"},"ruby":{"locked_version":"2.4.0","locked_source":{"sha256":"152fd0bd15a90b4a18213448f485d4b53e9f7662e1508190aa5b702446b29e3d","url":"https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.0.tar.gz"},"source_type":"url","described_version":"2.4.0","license":"BSD-2-Clause"},"rubygems":{"locked_version":"2.6.14","locked_source":null,"source_type":"project_local","described_version":"2.6.14","license":"MIT"},"bundler":{"locked_version":"1.12.5","locked_source":null,"source_type":"project_local","described_version":"1.12.5","license":"MIT"},"chef-gem-libyajl2":{"locked_version":"1.2.0","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"1.2.0","license":"Apache-2.0"},"chef-gem-ffi-yajl":{"locked_version":"2.3.4","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"2.3.4","license":"MIT"},"chef-gem-pkg-config":{"locked_version":"<skip>","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"<skip>","license":"LGPL-2.1"},"chef-gem-mini_portile2":{"locked_version":"2.1.0","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"2.1.0","license":"MIT"},"chef-gem-nokogiri":{"locked_version":"1.7.2","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"1.7.2","license":"MIT"},"chef-gem-ruby-prof":{"locked_version":"0.18.0","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"0.18.0","license":"BSD-2-Clause"},"chef-gem-byebug":{"locked_version":"11.0.1","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"11.0.1","license":"MIT"},"chef-gem-debug_inspector":{"locked_version":"0.0.3","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"0.0.3","license":"MIT"},"chef-gem-binding_of_caller":{"locked_version":"0.8.0","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"0.8.0","license":"MIT"},"chef-gem-rbnacl-libsodium":{"locked_version":"<skip>","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"<skip>","license":"MIT"},"chef-gem-bcrypt_pbkdf-ruby":{"locked_version":"<skip>","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"<skip>","license":"MIT"},"chef-gem-ffi":{"locked_version":"1.13.1","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"1.13.1","license":"BSD-3-Clause"},"chef-gem-json":{"locked_version":"2.3.1","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"2.3.1","license":"Ruby"},"chef-gem-ruby-shadow":{"locked_version":"2.5.0","locked_source":{"path":"/home/rmurphy/chef/omnibus/files/chef-gem"},"source_type":"path","described_version":"2.5.0","license":"Public-Domain"},"chef":{"locked_version":"local_source","locked_source":{"path":"/home/rmurphy/chef","options":{"exclude":["omnibus/vendor"]}},"source_type":"path","described_version":"local_source","license":"project_license"},"chef-appbundle":{"locked_version":"local_source","locked_source":{"path":"/home/rmurphy/chef/omnibus/files"},"source_type":"path","described_version":"local_source","license":"project_license"},"chef-cleanup":{"locked_version":"local_source","locked_source":{"path":"/home/rmurphy/chef/omnibus/files"},"source_type":"path","described_version":"local_source","license":"project_license"},"gem-permissions":{"locked_version":"0.0.1","locked_source":null,"source_type":"project_local","described_version":"0.0.1","license":"project_license"},"shebang-cleanup":{"locked_version":"0.0.2","locked_source":null,"source_type":"project_local","described_version":"0.0.2","license":"project_license"},"version-manifest":{"locked_version":"0.0.1","locked_source":null,"source_type":"project_local","described_version":"0.0.1","license":"project_license"},"openssl-customization":{"locked_version":null,"locked_source":{"path":"/home/rmurphy/chef/omnibus/files/openssl-customization"},"source_type":"path","described_version":null,"license":"project_license"},"chef-complete":{"locked_version":null,"locked_source":null,"source_type":"project_local","described_version":null,"license":"project_license"}},"build_version":"12.22.5+20201007160609","build_git_revision":"53c4a6cd48045d1de88a688cf719bb6760d3021f","license":"Apache-2.0"}' >> ~/chef/omnibus/pkg/version-manifest.json8) Create /opt/chef and /var/cache/omnibus and set the proper permissions. If either of these exist already, you should delete it in preparation for the new build.
sudo mkdir /opt/chef && sudo chown -R $USER:root /opt/chef
sudo mkdir /var/cache/omnibus && sudo chown -R $USER:root /var/cache/omnibus9) Build the custom Chef package from ~/chef/omnibus
cd ~/chef/omnibus && bundle exec omnibus build chef --log-level=internal --use-manifest=~/chef/omnibus/pkg/version-manifest.json10) Wait for the build to finish, this may take anywhere from 10-20 minutes. Once complete, the package will be located in ~/chef/omnibus/pkg
Troubleshooting
This is confirmed working in the below environment...
LSB Version: core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]
rvm 1.29.10 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
rake, version 12.3.0
Bundler version 1.17.3
Rubygems 3.0.8Seeing these warnings repeatedly seems to be harmless & expected during the build...
D | 2020-10-08T15:53:51+00:00 | ./include/ruby/intern.h:257:1: warning: ‘const’ attribute on function returning ‘void’ [-Wattributes]
D | 2020-10-08T15:53:51+00:00 | 257 | CONSTFUNC(void rb_error_untrusted(VALUE));
D | 2020-10-08T15:53:51+00:00 | | ^~~~~~~~~
D | 2020-10-08T15:53:51+00:00 | ./include/ruby/intern.h:259:1: warning: ‘const’ attribute on function returning ‘void’ [-Wattributes]
D | 2020-10-08T15:53:51+00:00 | 259 | CONSTFUNC(void rb_check_trusted(VALUE));
D | 2020-10-08T15:53:51+00:00 | | ^~~~~~~~~If I left anything out or you'd like to make any suggestions feel free to comment below.
Comments