JunOS SPACE installation notes

Installing JunOS SPACE can be a slog through documentation. These are my notes to help with the needed steps.

Assuming SPACE is going to be installed on VMWare, as version 16.1 or better, this is the recommended sizing:

32GB RAM (OVA installs as 8GB, increase it)
4 vCPU
~ 500GB storage space total

The OVA installs as about 250G storage space. 100G of that is /var. You’ll want to add to that. Depending on the size of the environment being managed and whether this node will also handle integrated logs, anywhere from 250GB to 1TB of additional space is appropriate. It’s possible to add 250GB to start with and then add additional space if required.

SPACE can be deployed as a cluster, all in the same subnet, as well as in a DR scenario across L3 boundaries. Most of my customers run a single instance, as their environment is not large enough to warrant cluster deployment and they can rely on VMWare for DR purposes.

SPACE requires two IP addresses – one for the physical node and one for the VIP, used for HTTPS GUI access. Any additional nodes would need one additional address in the same subnet.

SPACE can use a second interface to communicate with devices if desired. This can be handy if the device management interface and the GUI access need to be in separate subnets.

SPACE does not offer a supported way of firewalling itself. You’ll want to firewall it in your environment, at a minimum restricting access to internal subnets, better yet restricting access to trusted subnets. This is a list of the services used, subject to revision should I miss a few.

VIP:
– HTTPS inbound for GUI access. Optional Ping inbound.

Physical IP:
– ssh inbound for admin console access. Optional Ping inbound.
– DNS, NTP and SMTP outbound to your DNS/NTP/SMTP servers. Optional Ping outbound.
– HTTPS outbound to “*.juniper.net”, or if deploying as a customer instance connecting back to a Juniper partner, *.juniper.net and the partner SPACE proxy. When in doubt, this can be HTTPS outbound “to the Internet.” Optional Ping outbound.
– If you have a SPACE cluster with several nodes, they’ll communicate on that subnet using Multicast.

Physical IP or Device Management IP, if it was configured:
– ssh, ping and snmp outbound to device subnets
– snmp-trap inbound from device subnets

These are the parameters you should have before you install SPACE:

  • DNS server address
  • NTP server address
  • Time Zone
  • VIP IP
  • Physical IP
  • Gateway IP
  • If SPACE will be behind a NAT device for device access, you’ll need to specify that during setup
  • Node Name – for display in GUI, not the FQDN
  • admin password
  • maintenance password
  • super password
  • Desired FQDN – for DNS entry you will create
  • Cert for that FQDN – to avoid security warnings when connecting to GUI. This can absolutely be an internal trusted CA
  • Desired user authentication method – your options are Local, RADIUS and TACACS+
  • Juniper.net user name and password – for download of schemas and setup of ASAP, the pro-active service app
  • Any apps you will install such as ASAP (aka Service Now), Security Director, Network Director
  • If using Security Director or Network Director, license authorization keys for those and JS-PLATFORM. You’ll “cut” the license once SPACE is installed. If only using ASAP (ne Service Now), you will receive a permanent license once you connect SPACE to juniper.net, free of charge as long as you have at least one device under active maintenance.
  • SMTP server (and any authentication you need) for SPACE to send email
  • Username and password (or ssh key) you will use to manage devices. A “service account” such as “spcadmin” is often a good idea. This account needs “admin” rights in JunOS.
  • SNMPv3 or SNMPv2 read-only for use by SPACE. Optional (but recommended), allows Network Director and OpenNMS to monitor devices.

Once the OVA is installed, has been increased to 32GB RAM and has had an additional disk allocated to it – do NOT increase the size of the disk the OVA creates for you, that won’t work – you are ready to power it up and go through initial configuration. It’s pretty straightforward and asks for many of the parameters you gathered above.

The default username for console access is admin / abc123

The default username for GUI access is super / juniper123

When SPACE is up and running, you’ll have an option to expand virtual drive space. Choose it and assign all the additional disk space your VM guest received to /var. Be careful with this, if you accidentally assign it to another partititon (such as /, /tmp or /var/log), you’ll need to wipe the VM and start over.

Save your admin, maintenance and super passwords in an encrypted, centralized, backed up password safe. You can recover if you lose the admin password, but it takes access to the VMWare host and some effort.
“admin” is used for ssh access, “maintenance” is used for upgrades of the JunOS SPACE platform itself, and “super” is used for GUI access.

Additional housekeeping while on console and waiting for jboss and thus the GUI to start:

  • Change admin password expiration. Default is 70 days; you’ll likely want a longer timeout or “never”.
  • Change ssh session timeout. Default is 5 minutes. You can edit /etc/ssh/sshd_config and set ClientAliveInterval to 600 and ClientAliveCountMax to 3 and you’ll have 30 minutes.
  • Install VMWare Tools. Your VMWare admins will thank you.

If you want to avoid HTTPS security warnings when connecting to JunOS SPACE, create a DNS entry for its VIP address with the FQDN you chose, then create a cert (again, trusted internal CA is fine) for that. You’ll load that in the GUI under Administration -> CA/CRL certificates. Load the cert and any needed intermediate CAs.

Once in the GUI, you’ll want to change some of the default settings. Go to Administration -> Applications, right-click “Network Management Platform” and choose “Modify Application Settings”.
– “Allow Device Communication” is critical.
– “Add SNMP configuration to device for fault monitoring” can be useful if you want to use OpenNMS, but isn’t critical.
– “Configure commit synchronize” creates issues with single EX devices, uncheck that.
– “Manually resolve fingerprint conflicts” is probably more hassle than its worth for all but the most security-conscious customers.
– “Auto Resync”, “Approval workflow” and “commit confirmed” are useful
-Under “User”, set the timeout. 30 or 60 minutes seems reasonable for most environments.
– Under “Password”, set the password expiry in months. I’ve seen customers set this to “120” because they believe in the revised NIST guidelines and prefer good passwords over frequent changes.
– Under “Security”, the “Disable weak algorithms” checkbox will help the device pass an audit.

And hit “Modify”, wait for JunOS SPACE to restart its web server, and log back in.

If you are not going to use OpenNMS, you may disable it under Administration -> Applications -> Network Management Platform -> Manage Services

Under Administration -> DMI Schemas, set SPACE up to be able to pull DMI Schemas.
Click on the “Update Schema” icon, click the “SVN Repository” radio button and the “Configure” button. The URL is https://xml.juniper.net/dmi/repository/trunk/, the username and password are a juniper.net login that belongs to the organization running this SPACE instance. “Auto Install Schema” is a good idea as it avoids additional work. “Test Connection”, then “Save”.

Under Administration -> SMTP Servers, set up your mail server.

Under Administration -> Authentication Servers, set up your RADIUS/TACACS+ auth. I recommend “Remote-Local Authentication” so that you can still get into the unit using “super” if the remote authentication fails.

Under Administration -> Database Backup and Restore, you can set a backup to an scp server. It’s likely you’ll be relying on VMWare snapshots, but if you don’t have that in place, this is highly recommended.

Under Administration -> Purging Policy, set a policy to purge disk space periodically. Not really needed unless you take regular local DB backups or have very large device configuration files, in which case it becomes critical.

Under Administration -> CA/CRL certificates, install your HTTPS certificate.

Install any applications you’d like to use. ASAP (ne Service Now) is quite useful, and Security Director is the obvious choice for SRX policy management.

When deploying Security Director, I recommend also deploying a second node as Log Collector.  Unless you already have the SIEM IBM QRadar or Juniper JSA collecting logs, in which case you can just point Security Director towards those.
Log Collector will require another 16GB, 2 IP addresses (one in the same subnet as the main SPACE node for cluster comms and one for syslog, can be in the same subnet but need not be), and either 500GB of disk space and an NFS share, or 1TB (or more) of disk space to hold logs locally.

If you do use ASAP (Service Now), here are a few settings that’ll help you out:

You’ll add an “Organization”. If you are going through a partner proxy with PAR service instead of direct to Juniper, work with the partner on that setup. They’ll point your instance to their proxy, load a certificate file for their proxy, and they may set an auto-submit policy for incidents.

Administration -> Global Settings -> Core File Upload Configuration, set this to “Secure FTP upload through Service Now”. Otherwise devices will try to FTP directly to juniper.net and that will likely fail.

Here are some good videos by Juniper on using ASAP:

Video #1: https://www.youtube.com/watch?v=EM2w86T96Ac
Video #2: https://www.youtube.com/watch?v=HiAKA2ItROg
Video #3: https://www.youtube.com/watch?v=gU-f1hxttCY
Video #4: https://www.youtube.com/watch?v=a9mUSmJXST4

 

Lastly, when adding devices to SPACE, consider assigning them a public tag such as “All Devices” and configuring Configuration Files backup to act on that tag on a schedule, say once a day.

You can create schedules to find new devices automatically, and you can of course use the base JunOS SPACE application to upgrade firmware and make bulk configuration changes.

Advertisements

JunOS SPACE upgrade to 16.1r2

These are my notes for upgrading JunOS SPACE from 15.2r2 to 16.1r1 or 16.1r2. They are meant to be consumed together with Juniper’s upgrade instructions. Since you are installing a fresh copy of JunOS SPACE as part of this upgrade, maybe now is also a good time to revisit some default settings.

  • 16.1 is the first release where the default partition sizes in the OVA are “sane”. The only partition you’ll need to add to is /var. It is 100GB large by default. An additional 250GB is fine for most installations; large installations with massive DBs might want as much as an additional 1TB.
  • You may not have enough space on the disk to take a backup using the 15.2r2 backup patch as long as OpenNMS remains enabled. In that case, disable it; then after reinstall and import, take additional steps to re-enable it. Disabling OpenNMS is done from Administration -> Applications -> Network Management Platform -> right-click and Manage Services
  • When taking the backup, I then opted not to backup PGSQL (that’s OpenNMS) and FMPM (since I happen not to have any FMPM nodes). This reduced the size to something manageable.
  • You may need the ServiceNow image file when taking the backup. If so, copy it to SPACE using command-line scp, then move it to /var/cache/jboss/jmp/Service-Now.VERSION (the backup process will tell you the exact location), and hit Enter to let the backup continue. For Service Now 16.1r1, the location is /var/cache/jboss/jmp/Service-Now.16.1R1.15
  • You will require an external scp server to copy the backup file to, or you can use a USB stick with FAT32 (no more than 32GB) if upgrading a JA2500 appliance.
  • Front USB is detected as /dev/sdb, use dmesg to make sure. Then mount:
    mkdir /tmp/pendrive
    mount -t vfat /dev/sdb1 /tmp/pendrive
    You can check with fdisk -l
  • To restore from USB, go through initial configuration.  When you come to restore choice (Remote, USB, Local), ssh to device and mount -t vfat /dev/sdb1 /tmp/pendrive, then use serial console to choose USB
  • If you don’t have an scp server, you can choose 127.0.0.1 during backup; then copy the file over to the new server during restore and choose “Local”
  • You will need these things to configure your new SPACE instance:
    DNS server
    NTP server
    TZ
    VIP IP
    Phys1 IP
    Phys2-N IP
    GW IP
    NodeName1
    NodeName2-N
    License File
    List of Apps
    admin password
    maintenance password
  • After restoration, adding space to /var etc, check the settings for Network Management Platform. “Allow device communication” may be off. Turn it on so devices will move to “Up” status.
  • For a secondary node, it doesn’t ask for NTP on initial setup. Set this and TZ manually. Once secondary node is up, you’ll need to add it from GUI as well.
  • Don’t forget chage admin and the ClientAliveInterval / ClientAliveCountMax in /etc/ssh/sshd_config
  • If you disabled OpenNMS before the backup, it won’t start after import. This is how you get it back in a default state.
    Disable OpenNMS from GUI
    service postgresql-9.4 status
    If it’s down: service postgresql-9.4 start
    Now to create the DB:
    service jmp-watchdog stop
    service jmp-opennms stop
    For the following, passwords are postgres and opennms respectively
    psql -U postgres -c ‘ALTER ROLE opennms SUPERUSER’
    psql -U opennms postgres -c ‘drop database opennms;’
    psql -U opennms postgres -c “create database opennms encoding ‘unicode'”
    psql -U postgres -c ‘ALTER ROLE opennms NOSUPERUSER’
    /opt/opennms/bin/install -dis
    service jmp-watchdog start
    Then enable OpenNMS from GUI

 

JunOS SPACE fails when upgrading applications

When upgrading JunOS SPACE, the applications installed on it need to be upgraded as well. I’ve seen jboss crash and restart when several applications are upgraded in a row. This happened after moving to SPACE 16.1r2 and then again when upgrading to 17.1r1.

JTAC advised that the issue is that Java runs out of “PermGen” memory. Assuming that SPACE is installed on a JA2500 or a VM with 32GB of memory, these changes should resolve the issue. They may have to be reapplied after each upgrade of the core SPACE software.

  • stop jboss and watchdog
    service jmp-watchdog stop
    service jmp-watchdog status (make sure it has stopped)
    service jboss stop
    service jboss status (make sure it has stopped)
  • edit /var/jboss/domain/configuration/host.xml.slave and change
    <option value=”-XX:MaxPermSize=512m”/>
    to
    <option value=”-XX:MaxPermSize=1025m”/>
  • start watchdog
    service jmp-watchdog start
    service jmp-watchdog status

    The watchdog process will restart jboss so there is no need to restart jboss manually.

 

Installing VMWare Tools (Open VM Tools) on JunOS SPACE 16.1 or newer

These instructions are for JunOS SPACE 16.1 or newer. I also have instructions for JunOS SPACE 15.2 or older.

JunOS SPACE, Juniper’s management tool for JunOS devices (switches, routers, firewalls), officially supports Open VM Tools for management from ESXi. Unfortunately, Juniper’s instructions are to build Open VM Tools, and that won’t work without a dev environment, which is not present in JunOS SPACE.

SPACE 16.1 is built on CentOS 6, which in turn is a RHEL 6 clone. Open VM Tools exist in CentOS 6 repositories, so all we need to do is to enable those repositories and we can install binaries.

Install

Navigate to /etc/yum.repos.d and create a new file named centos6.repo, with this content:

[centos6]
name=CentOS 6 Repository
baseurl=http://mirror.centos.org/centos/6/os/$basearch
enabled=1
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/6/os/x86_64/RPM-GPG-KEY-CentOS-6

[extras]
name=CentOS 6 Extras Repository
baseurl=http://mirror.centos.org/centos/6/extras/$basearch
enabled=1
gpgcheck=0

Back on command line, add the EPEL repository:

yum install epel-release

Install Open VM Tools:

yum install open-vm-tools

Start them:

service vmtoolsd start

Cleanup

For good measure, you can now disable the centos 6 and epel repos again, by editing centos6.repo and epel.repo in /etc/yum.repos.d/ and setting this line for centos6, extras, and epel:

enabled=0

Verify those repos are disabled:

yum repolist

 

JunOS SPACE upgrade hangs at 0%

I attempted to upgrade a JunOS SPACE instance from 15.2R1 to 15.2R2. It would sit at “upgrade process has not started” and 0%. If I changed the URI to the base, I’d be back in the SPACE GUI as if nothing had happened and I had never entered maintenance mode.

This was caused by a failed upgrade months earlier which left a msg.<date> file behind in /var/jmp_upgrade/master/msg . Deleting that file allowed me to successfully upgrade the unit.

After a successful upgrade, the msg/ directory will be empty in both the master and slave directories.

In the process, I learned about a few more files that SPACE looks for. If these exist from a failed upgrade, they can keep a new upgrade from starting. Delete these if they exist:

/var/log/activeUpgradeStatus.log

/var/jmp_upgrade/slave/log/upgradeMetaData.txt

You can find a clue as to why your upgrade is not proceeding in these two directories:

/var/jmp_upgrade/slave/log/

/var/jmp_upgrade/master/log/

Look for log files named after the current and target SPACE version.

Also monitor this file for any issues with maintenance mode:

/tmp/maintenance.log

 

Change admin user password expiry on JunOS SPACE

The admin user on JunOS SPACE, which is used for ssh / root access, has a default password expiry of 70 days. This may not be desired.

NB: An upgrade of the JunOS SPACE platform will set the admin password expiry to the default of 70 days again. To avoid the admin user password being prompted for change after the upgrade, this procedure needs to become part of your documented upgrade procedure for JunOS SPACE:

  • Change admin user last password change to be today using chage
  • Upgrade SPACE
  • Change admin user expiry to “never” using chage

The Linux command “chage” will show you the current settings for a user and let’s you change those:

  • Log in as admin via ssh
  • Choose “(Debug) run shell”
  • Use “chage” to see and then change the admin password expiry:
    chage -l admin
    Last password change : Mar 15, 2016
    Password expires : May 24, 2016
    Password inactive : never
    Account expires : never
    Minimum number of days between password change : 7
    Maximum number of days between password change : 70
    Number of days of warning before password expires : 7
  • Change the parameters: 

    chage admin
    Changing the aging information for admin
    Enter the new value, or press ENTER for the default

    Minimum Password Age [7]: 0
    Maximum Password Age [70]: 99999
    Last Password Change (YYYY-MM-DD) [2016-03-15]: <today’s date>
    Password Expiration Warning [7]:
    Password Inactive [-1]:
    Account Expiration Date (YYYY-MM-DD) [1969-12-31]:

    And verify:

    chage -l admin
    Last password change : Mar 25, 2016
    Password expires : never
    Password inactive : never
    Account expires : never
    Minimum number of days between password change : 0
    Maximum number of days between password change : 99999
    Number of days of warning before password expires : 7

Reset admin password on JunOS SPACE VM

Picture this: Bob, your JunOS SPACE administrator, left the company. IT diligently wiped his laptop. Bob was the only one who had the admin password (for ssh / CLI access) to your JunOS SPACE installation.

After vowing to do better in future and storing all infrastructure passwords in some form of centralized, encrypted, backed-up password safe, you are left with the task of restoring access.

If you are running your JunOS SPACE instance on an appliance, all you need is a USB stick and Juniper’s instructions. (Note: That may not be entirely true – see discussion of how /etc/shadow behaves in 14.1R2 of SPACE, below)

If you are running your JunOS SPACE instance in a VM, read on.

You’ll need vSphere-level access to the VMWare host JunOS SPACE runs on. Make friends with the VMWare admin.

You’ll need an ISO of a Linux rescue disk.

Power down the SPACE VM and change its boot properties to force boot into BIOS, and delay by 10,000ms for good measure, like so:

Screenshot 2015-07-16 10.44.33

Power on the SPACE VM, and open a Console. You’ll find yourself in the BIOS shortly. Connect the virtual CD drive to the ISO you downloaded (that’s the CD-with-wrench icon in the Console) and change the BIOS to boot from CD first, like so:

Screenshot 2015-07-16 10.54.02

Alternatively, you could have left the BIOS alone and hit ESC to get the boot menu during the 10 second boot delay.

Whichever option you choose, make sure the CD has finished connecting (loading the ISO) before you hit F10/Enter. If you are remote to your VMWare host, it may make sense to upload the ISO to the datastore first, and connect the virtual CD to that copy.

Once the VM has booted from the rescue ISO, I chose the “standard with US keymap” boot option for the rescue disk.

Use lvscan to see the device names of the SPACE disks:

 ACTIVE '/dev/jmpvgnocf/lvroot' [25.41 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvtmp' [17.91 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvvar' [207.78 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvlog' [4.88 GB] inherit

Mount the root partition:

mount /dev/jmpvgnocf/lvroot /mnt/custom

Move the shadow- backup file (don’t delete it):

mv /mnt/custom/etc/shadow- /mnt/custom/root

I tried this without moving shadow- and it would revert to the previous admin password after editing shadow.

Juniper’s original instructions for the appliance, based on the now-ancient SPACE 1.2, want you to save an empty password (::) for admin in shadow. When I tried that, I could no longer log in.

Instead, I used passwd on the rescue CD to get the crypt string for the password “abc123”. It is:

$6$MsjX3IGJ$KVp.m.rQDK5PG1ELcjNajJUbBttXZZzXY/mtQhmNG4PmqSyuVYXgcOjuwjksXkz1rf5DM.laraWKpNf9.T/mp0

With this in hand, I could “vi /mnt/custom/etc/shadow” and insert the string for the new password, which looks like this:

Screenshot 2015-07-16 12.55.28

Save that file using “:x!” in vi and you are ready to “reboot”

Note: Typing that string in the VMWare Console is going to be extremely error-prone. It makes sense to ssh to the rescue Linux instead, so you can copy/paste. Use “ifconfig” to see whether you received a DHCP address. If not, you’ll need to follow the instructions for the rescue image to enable networking – or manually type the password string.

If you changed the BIOS to boot from CDROM first, undo that change when the boot screen comes up.

After reboot, you should be able to log in as admin with the default password and use the option “1” to change the password:

Screenshot 2015-07-16 12.54.59

Now that you’re back in the CLI, you can follow Juniper’s instructions for resetting the password for the ‘super’ user which is used for GUI access and the ‘maintenance’ user which is used for software updates.

With those three accounts restored, you have the access needed to administrate JunOS SPACE again – and set up further GUI authentication for users as desired.