JunOS SPACE upgrade to 16.1r2

These are my notes for upgrading JunOS SPACE from 15.2r2 to 16.1r1 or 16.1r2. They are meant to be consumed together with Juniper’s upgrade instructions.

  • 16.1 is the first release where the default partition sizes in the OVA are “sane”. The only partition you’ll need to add to is /var. It is 100GB large by default. An additional 250GB is fine for most installations; large installations with massive DBs might want as much as an additional 1TB.
  • You may not have enough space on the disk to take a backup using the 15.2r2 backup patch as long as OpenNMS remains enabled. In that case, disable it; then after reinstall and import, take additional steps to re-enable it. Disabling OpenNMS is done from Administration -> Applications -> Network Management Platform -> right-click and Manage Services
  • When taking the backup, I then opted not to backup PGSQL (that’s OpenNMS) and FMPM (since I happen not to have any FMPM nodes). This reduced the size to something manageable.
  • You may need the ServiceNow image file when taking the backup. If so, copy it to SPACE using command-line scp, then move it to /var/cache/jboss/jmp/Service-Now.VERSION (the backup process will tell you the exact location), and hit Enter to let the backup continue. For Service Now 16.1r1, the location is /var/cache/jboss/jmp/Service-Now.16.1R1.15
  • You will require an external scp server to copy the backup file to, or you can use a USB stick with FAT32 (no more than 32GB) if upgrading a JA2500 appliance.
  • Front USB is detected as /dev/sdb, use dmesg to make sure. Then mount:
    mkdir /tmp/pendrive
    mount -t vfat /dev/sdb1 /tmp/pendrive
    You can check with fdisk -l
  • To restore from USB, go through initial configuration.  When you come to restore choice (Remote, USB, Local), ssh to device and mount -t vfat /dev/sdb1 /tmp/pendrive, then use serial console to choose USB
  • If you don’t have an scp server, you can choose 127.0.0.1 during backup; then copy the file over to the new server during restore and choose “Local”
  • You will need these things to configure your new SPACE instance:
    DNS server
    NTP server
    TZ
    VIP IP
    Phys1 IP
    Phys2-N IP
    GW IP
    NodeName1
    NodeName2-N
    License File
    List of Apps
    admin password
    maintenance password
  • After restoration, adding space to /var etc, check the settings for Network Management Platform. “Allow device communication” may be off. Turn it on so devices will move to “Up” status.
  • For a secondary node, it doesn’t ask for NTP on initial setup. Set this and TZ manually. Once secondary node is up, you’ll need to add it from GUI as well.
  • Don’t forget chage admin and the ClientAliveInterval / ClientAliveCountMax in /etc/ssh/sshd_config
  • If you disabled OpenNMS before the backup, it won’t start after import. This is how you get it back in a default state.
    Disable OpenNMS from GUI
    service postgresql-9.4 status
    If it’s down: service postgresql-9.4 start
    Now to create the DB:
    service jmp-watchdog stop
    service jmp-opennms stop
    For the following, passwords are postgres and opennms respectively
    psql -U postgres -c ‘ALTER ROLE opennms SUPERUSER’
    psql -U opennms postgres -c ‘drop database opennms;’
    psql -U opennms postgres -c “create database opennms encoding ‘unicode'”
    psql -U postgres -c ‘ALTER ROLE opennms NOSUPERUSER’
    /opt/opennms/bin/install -dis
    service jmp-watchdog start
    Then enable OpenNMS from GUI

 

JunOS SPACE fails when upgrading applications

When upgrading JunOS SPACE, the applications installed on it need to be upgraded as well. I’ve seen jboss crash and restart when several applications are upgraded in a row. This happened after moving to SPACE 16.1r2 and then again when upgrading to 17.1r1.

JTAC advised that the issue is that Java runs out of “PermGen” memory. Assuming that SPACE is installed on a JA2500 or a VM with 32GB of memory, these changes should resolve the issue. They may have to be reapplied after each upgrade of the core SPACE software.

  • stop jboss and watchdog
    service jmp-watchdog stop
    service jmp-watchdog status (make sure it has stopped)
    service jboss stop
    service jboss status (make sure it has stopped)
  • edit /var/jboss/domain/configuration/host.xml.slave and change
    <option value=”-XX:MaxPermSize=512m”/>
    to
    <option value=”-XX:MaxPermSize=1025m”/>
  • start watchdog
    service jmp-watchdog start
    service jmp-watchdog status

    The watchdog process will restart jboss so there is no need to restart jboss manually.