@ -1,52 +1,40 @@
# Upgrading Development Zones
Over time, we update the base development environment that everyone is using.
The current target is the x86_64 2018 .4.x image series as noted in
The current target is the x86_64 202 1.4.x image series as noted in
[the SmartOS Getting Started Guide ](../README.md#importing-the-zone-image ).
The purpose of this guide is to describe how an **existing** development zone
should be upgraded from one version of pkgsrc to the next. In the past, because
pkgsrc versions were compatible, we've been able to upgrade devzones in
place (e.g. use pkgin and pkg_* utilities to upgrade from 2015.4.x -> 2016.4)
(x86_64 2018.4) should be upgraded from one version of pkgsrc to the next
(x86_64 2021.4). In the past, the upgrade for this target was more disruptive
than previous ones, as we went from a `multiarch` release to an `x86_64`
release, which needed an entirely new /opt/local installation as pkgsrc
cannot upgraded across this sort of boundary.
The upgrade for this target is more disruptive than previous ones, as we're
going from a `multiarch` release to an `x86_64` release, which need an entirely
new /opt/local installation as pkgsrc cannot be upgraded across this sort of
boundary.
The simplest and safest route is to simply create a [fresh
The simplest and safest route still is to simply create a [fresh
zone](../README.md#setting-up-a-build-environment)
However, if you have customisations in your zone, reinstalling may be
time-consuming. This guide documents a procedure to allow you to upgrade
without provisioning a new devzone. Unfortunately due to the nature of the
upgrade, it cannot be complete for every use case, and you will likely need
to take additional steps to preserve and restore your configuration.
without provisioning a new devzone.
If you have advice on what else we could add to these instructions, please
do get in touch.
## Preparing to Upgrade
NOTE: All pkgin and pkg_add instructions below should be done as root, or
with pfexec, or with sudo.
### Snapshot package list
First, it's helpful to snapshot the package list that you have
installed. You should do this by running:
```
pkg_info -u | cut -d' ' -f1 | sort > /package.list
```
Aside: normally, we would produce this list using:
```
pkgin export | sort > /package.list
```
but we believe a bug in pkgin from 2016 causes the resulting list to
be empty. That bug has been fixed in the version of pkgsrc we're upgrading
to in these instructions.
### Cleaning up existing builds
Because all the base pkgsrc libraries that are used are going to
@ -80,11 +68,11 @@ In this case, we're interested in upgrading the instance called
`march-dev` . Next we create a snapshot and verify it exists:
```
$ triton inst snapshot create --name=2018 .4-upgrade march-dev
Creating snapshot 2018 .4-upgrade of instance march-dev
$ triton inst snapshot create --name=202 1.4-upgrade march-dev
Creating snapshot 202 1.4-upgrade of instance march-dev
$ triton inst snapshot list march-dev
NAME STATE CREATED
2018 .4-upgrade created 2018-09-28T18:40:14.000Z
202 1.4-upgrade created 2018-09-28T18:40:14.000Z
```
#### Manual Snapshots in SmartOS
@ -97,8 +85,8 @@ Then you use the `create-snapshot` option.
[root@00-0c-29-37-80-28 ~]# vmadm list
UUID TYPE RAM STATE ALIAS
79809c3b-6c21-4eee-ba85-b524bcecfdb8 OS 4096 running multiarch
[root@00-0c-29-37-80-28 ~]# vmadm create-snapshot 79809c3b-6c21-4eee-ba85-b524bcecfdb8 2018 .4-upgrade
Created snapshot 2018 .4-upgrade for VM 79809c3b-6c21-4eee-ba85-b524bcecfdb8
[root@00-0c-29-37-80-28 ~]# vmadm create-snapshot 79809c3b-6c21-4eee-ba85-b524bcecfdb8 202 1.4-upgrade
Created snapshot 202 1.4-upgrade for VM 79809c3b-6c21-4eee-ba85-b524bcecfdb8
```
If your VM has delegated snapshots, you won't be able to use `vmadm` to take
@ -120,7 +108,7 @@ development zone. If you encounter problems, please don't hesitate to
reach out for assistance.
The approach we describe is to cleanly shutdown services that
are running from /opt/local, move /opt/local aside, install the 2018 Q4
are running from /opt/local, move /opt/local aside, install the 202 1Q4
x86-64 pkgsrc bootstrap bundle. We then reinstall as many packages as
possible from the set that was previously manually installed, noting that
some packages may have been dropped from the pkgsrc repository.
@ -177,89 +165,91 @@ be used to determine modified properties. `xmllint` may also be used to
produce a view of a given pair of XML files in order to more easily compare
them.
### Clean up pkgin database
### Update pkgin configuration from 2018.4 to 2021.4
NOTE: This step is revertable if no subsequent steps are taken.
Edit these files:
```
/opt/local/etc/pkg_install.conf
/opt/local/etc/pkgin/repositories.conf
```
And change any instance of `2018Q4` to `2021Q4` . There should be one
instance in each file. Here's is a pre-upgrade view:
```
smartos-build(~)[0]% grep Q4 /opt/local/etc/pkg_install.conf
PKG_PATH=https://pkgsrc.joyent.com/packages/SmartOS/2018Q4/x86_64/All
smartos-build(~)[0]% grep Q4 /opt/local/etc/pkgin/repositories.conf
https://pkgsrc.joyent.com/packages/SmartOS/2018Q4/x86_64/All
smartos-build(~)[0]%
```
The pkgin repository database needs to be removed. To do that you will
need to run the following command:
and a post-upgrade view:
```
rm -rf /var/db/pkgin
smartos-build-2(~)[0]% grep Q4 /opt/local/etc/pkg_install.conf
PKG_PATH=https://pkgsrc.joyent.com/packages/SmartOS/2021Q4/x86_64/All
smartos-build-2(~)[0]% grep Q4 /opt/local/etc/pkgin/repositories.conf
https://pkgsrc.joyent.com/packages/SmartOS/2021Q4/x86_64/All
smartos-build-2(~)[0]%
```
### Move aside /opt/local and bootstrap the new pkgsrc install
### Manually install necessary prerequisite upgrades
The instructions in this step are an abbreviated version of the [pkgsrc illumos
installation](http://pkgsrc.joyent.com/install-on-illumos/) instructions.
NOTE: This step is NOT revertable once taken.
Note that the old /opt/local directory is saved to /opt/local.bak
A few upgrades first need to be installed explicitly, to prevent dependency
tripping:
```
cd /var/tmp
curl -O https://pkgsrc.joyent.com/packages/SmartOS/bootstrap/bootstrap-2018Q4-x86_64.tar.gz
mv /opt/local/ /opt/local.bak
tar xzf bootstrap-2018Q4-x86_64.tar.gz -C /
pkg_add -U pkgin
pkg_add -U libarchive pkg_install pkgin
```
### Reinstall packages
Those will enable a 2021.4-savvy pkgin to perform the next step.
Now that we've bootstrapped, we'd like to reinstall packages that were
previously installed. Noting that these may not map exactly, we prune the
old version numbers, and install the latest versions of each package.
### Perform a full upgrade
Slightly complicating things, is the fact that the Illumos build needs
specific versions of certain packages. We'll avoid re-installing packages
in that set and let the smartos-live `configure` script install them for
us instead.
Now that we've bootstrapped, we'd like to upgrade.
```
LIST=""
AVOID_PKGS="dmake sgstools rpcgen astmsgtools"
for package in $(/bin/cat /package.list); do
VER=$(echo $package | /bin/awk -F- '{print $NF}')
PKG=$(echo $package | /bin/sed -e "s/-$VER$//g")
avoid=''
for avoid_pkg in $AVOID_PKGS; do
if [[ "$avoid_pkg" == "$PKG" ]]; then
avoid=true
fi
done
if [[ -n "$avoid" ]]; then
continue
fi
LIST="$LIST $PKG"
done
pkgin in $LIST
pkgin upgrade
```
At this point you will see output similar to :
The output should look like this:
```
calculating dependencies...done.
smartos-build-2(~)[0]% pfexec pkg_add -U libarchive pkg_install pkgin
===========================================================================
The following directories are no longer being used by openssl-1.0.2p,
and they can be removed if no other packages are using them:
133 packages to install:
png-1.6.36 libidn-1.34 libxcb-1.13.1 libXdmcp-1.1.2nb1 libXau-1.0.8nb1 libXrender-0.9.10nb1 freetype2-2.9.1nb1 fontconfig-2.13.1 xmlcatmgr-2.2nb1 p5-Net-SSLeay-1.85nb1
p5-Net-LibIDN-0.12nb10 p5-Mozilla-CA-20180117nb1 p5-Socket6-0.29 p5-Net-IP-1.26nb6 p5-MIME-Base64-3.15nb4 p5-IO-Socket-INET6-2.72nb4 p5-Digest-MD5-2.55nb3 mit-krb5-1.16.2
tcl-8.5.19 libXft-2.3.2nb2 libXext-1.3.3nb1 libX11-1.6.7 libunistring-0.9.10 libxml2-2.9.9 p5-GSSAPI-0.28nb9 p5-Digest-HMAC-1.03nb8 p5-Net-Domain-TLD-1.75nb2 p5-Net-DNS-1.19
p5-IO-CaptureOutput-1.11.04nb4 p5-TimeDate-2.30nb5 p5-IO-Socket-SSL-2.060 tk-8.6.9 py27-pytz-2018.7 libfontenc-1.1.3nb1 db4-4.8.30 tcp_wrappers-7.6.4 libiconv-1.14nb3
libffi-3.2.1nb4 nghttp2-1.35.1nb2 libssh2-1.8.0 libidn2-2.0.5 perl-5.28.1 pcre2-10.32 p5-Net-SMTP-SSL-1.04nb2 p5-MailTools-2.20nb1 p5-Error-0.17027 p5-Email-Valid-1.202nb2
p5-Authen-SASL-2.16nb6 libtool-info-2.4.6 libtool-fortran-2.4.6nb1 libtool-base-2.4.6nb2 git-gitk-2.20.1 git-contrib-2.20.1 libgpg-error-1.33 py27-setuptools-40.6.3
py27-babel-2.6.0 mkfontscale-1.1.3 mkfontdir-1.0.7 encodings-1.0.4nb1 libXfixes-5.0.3nb1 openldap-client-2.4.47 cyrus-sasl-2.1.27 gettext-lib-0.19.8.1 mpfr-4.0.1 gmp-6.1.2
python27-2.7.15nb1 expat-2.2.6 libuuid-2.32.1 liblognorm-2.0.5 libfastjson-0.99.8 libestr-0.1.10 curl-7.64.0 libuv-1.24.1 libcares-1.15.0 icu-63.1nb2 http-parser-2.8 .1
pcre-8.42 gettext-tools-0.19.8.1nb1 gettext-m4-0.19.8.1nb1 gettext-asprintf-0.19.8.1 pkgconf-1.4.1nb1 m4-1.4.18nb1 libtool-2.4.6 gmake-4.2.1nb1 git-docs-2.20.1 git-base-2.20. 1
gcc7-7.3.0nb4 bison-3.0.4nb4 binutils-2.26.1nb1 automake-1.16.1 autoconf-2.69nb8 libltdl-2.4.6 git-2.20.1 libgcrypt-1.8.4 py27-genshi-0.7 zip-3.0nb3 unzip-6.0nb8
libXtst-1.2.3nb1 libXi-1.7.9nb1 dejavu-ttf-2.37 libpsl-0.20.2nb2 sudo-1.8.26 gtar-base-1.30 smtools-20160926 zoneinit-1.6.9 gawk-4.2.1 py27-expat-2.7.15 rsyslog-8.38.0 nb1
findutils-4.6.0nb2 coreutils-8.29nb1 patch-2.7.6nb1 nodejs-10.14.2nb1 gsed-4.6 grep-3.1nb2 pigz-2.4 cdrtools-3.01nb1 py27-sqlite3-2.7.15nb14 gettext-0.19.8 .1
build-essential-1.3 postfix-3.3.2 squid-3.5.28nb1 diffutils-3.6 scmgit-2.0 flex-2.6.4 less-530 libxslt-1.1.32nb1 nasm-2.14 manifold-0.2.0 openjdk7-1.7.141nb9
wget-1.20.1 p5-XML-Parser-2.44nb4 changepass-1.3.3
/opt/local/etc/openssl/certs
libgpg-error-1.43 libgcrypt-1.9.4 libfontenc-1.1.4 libffi-3.4.2nb1
libfastjson-0.99.8nb1 libestr-0.1.11 libcares-1.18.1 libXi-1.8 libXft-2.3.4
libXext-1.3.4 libXdmcp-1.1.3 libXau-1.0.9 libX11-1.7.3.1 less-563 lcms2-2.12
jbigkit-2.1nb1 icu-70.1 http-parser-2.9.4 harfbuzz-3.1.2nb1 gtar-base-1.34
gsed-4.8nb1 grep-3.7 gmp-6.2.1nb2 gmake-4.3nb3 glib2-2.70.2nb1
git-gitk-2.34.1 git-2.34.1 giflib-5.2.1nb4 gettext-lib-0.21
gettext-asprintf-0.21 gettext-0.21 gcc7-libs-7.5.0nb5 gcc7-7.5.0nb5
gawk-5.1.1 freetype2-2.10.4 fontconfig-2.13.1nb5 findutils-4.8.0 expat-2.4 .1
encodings-1.0.5 emacs26-nox11-26.3nb1 diffutils-3.7 db4-4.8.30nb 1
cyrus-sasl-2.1.27nb2 curl-7.81.0 coreutils-9.0 cdrtools-3.02a10
bmake-20200524nb1 bison-3.8.2 binutils-2.37 automake-1.16.5 autoconf-2.71 nb1
python27-2.7.18nb6 perl-5.34.0nb3 git-docs-2.34.1 git-contrib-2.34 .1
git-base-2.34.1nb1 gettext-tools-0.21nb3 gettext-m4-0.21
p5-Net-SSLeay-1.90nb1 pcre2-10.39
0 to refresh, 0 to upgrade, 132 to install
357M to download, 1009M to install
7 packages to install:
libXScrnSaver-1.2.3 brotli-1.0.9 blas-3.10.0 gcc10-10.3.0 lmdb-0.9.29
graphite2-1.3.14 python39-3.9.9nb1
proceed ? [Y/n]
.
.
.
26 to refresh, 125 to upgrade, 7 to install
726M to download, 595M to install
proceed ? [Y/n]
```
After the install has completed, you should review the install output,
@ -281,7 +271,7 @@ Recall that before upgrading, we saved a list of old SMF manifests in
on your new /opt/local pkgsrc installation.
If those manifests do not exist, then it's likely that the corresponding
package does not exist in the 2018 Q4 pkgsrc install, and that attempting to
package does not exist in the 202 1Q4 pkgsrc install, and that attempting to
re-enable the SMF service post-upgrade will fail.
In that case, the SMF service should be deleted using:
@ -309,24 +299,25 @@ but this warning can be ignored.
Finally, we can compare which packages are now installed:
```
pkg_info -u | cut -d' ' -f1 | sort > /package.list.new
pkgin export | sort > /package.list.new
/opt/local/bin/diff -y /package.list /package.list.new
```
Note that the packages normally installed by smartos-live's `configure`
script will be missing at this point. When you next run `configure` in
script might be missing at this point. When you next run `configure` in
advance of doing a smartos-live build, they will be installed from
http://us-east.manta.joyent.com/Joyent_Dev/public/releng/pkgsrc.
At this point, you should be able to reboot your dev zone and have it
come up cleanly. Note that the following files in /etc will now lie to
you:
At this point, you should be able to build a post-OS-8349 (2021.4) revision
of smartos-live and repos. NOTE that illumos-extra must be updated
concurrently with smartos-live. You may also reboot your dev zone and have it
come up cleanly. Note that the following files in /etc will now lie to you:
* /etc/motd
* /etc/pkgsrc_version
You may find it useful to manually update those files to correspond to
the /opt/local 2018 Q4 pkgsrc installation.
the /opt/local 202 1Q4 pkgsrc installation.
## Testing
@ -340,10 +331,6 @@ $ ./configure && gmake live
$
```
Note that during the `configure` phase, if gcc49 does not exist on the
system, it will be installed as it's still needed for bootstrapping the
`proto.strap` gcc compiler used by the build.
## Cleaning Up
Once you're satisfied, you should go through and delete the snapshots