Installing VMWare Tools (Open VM Tools) on JunOS SPACE 16.1 or newer

These instructions are for JunOS SPACE 16.1 or newer. I also have instructions for JunOS SPACE 15.2 or older.

JunOS SPACE, Juniper’s management tool for JunOS devices (switches, routers, firewalls), officially supports Open VM Tools for management from ESXi. Unfortunately, Juniper’s instructions are to build Open VM Tools, and that won’t work without a dev environment, which is not present in JunOS SPACE.

SPACE 16.1 is built on CentOS 6, which in turn is a RHEL 6 clone. Open VM Tools exist in CentOS 6 repositories, so all we need to do is to enable those repositories and we can install binaries.

Install

Navigate to /etc/yum.repos.d and create a new file named centos6.repo, with this content:

[centos6]
name=CentOS 6 Repository
baseurl=http://mirror.centos.org/centos/6/os/$basearch
enabled=1
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/6/os/x86_64/RPM-GPG-KEY-CentOS-6

[extras]
name=CentOS 6 Extras Repository
baseurl=http://mirror.centos.org/centos/6/extras/$basearch
enabled=1
gpgcheck=0

Back on command line, add the EPEL repository:

yum install epel-release

Install Open VM Tools:

yum install open-vm-tools

Start them:

service vmtoolsd start

Cleanup

For good measure, you can now disable the centos 6 and epel repos again, by editing centos6.repo and epel.repo in /etc/yum.repos.d/ and setting this line for centos6, extras, and epel:

enabled=0

Verify those repos are disabled:

yum repolist

 

Advertisements

JunOS SPACE upgrade hangs at 0%

I attempted to upgrade a JunOS SPACE instance from 15.2R1 to 15.2R2. It would sit at “upgrade process has not started” and 0%. If I changed the URI to the base, I’d be back in the SPACE GUI as if nothing had happened and I had never entered maintenance mode.

This was caused by a failed upgrade months earlier which left a msg.<date> file behind in /var/jmp_upgrade/master/msg . Deleting that file allowed me to successfully upgrade the unit.

After a successful upgrade, the msg/ directory will be empty in both the master and slave directories.

In the process, I learned about a few more files that SPACE looks for. If these exist from a failed upgrade, they can keep a new upgrade from starting. Delete these if they exist:

/var/log/activeUpgradeStatus.log

/var/jmp_upgrade/slave/log/upgradeMetaData.txt

You can find a clue as to why your upgrade is not proceeding in these two directories:

/var/jmp_upgrade/slave/log/

/var/jmp_upgrade/master/log/

Look for log files named after the current and target SPACE version.

Also monitor this file for any issues with maintenance mode:

/tmp/maintenance.log

 

Change admin user password expiry on JunOS SPACE

The admin user on JunOS SPACE, which is used for ssh / root access, has a default password expiry of 70 days. This may not be desired.

NB: An upgrade of the JunOS SPACE platform will set the admin password expiry to the default of 70 days again. To avoid the admin user password being prompted for change after the upgrade, this procedure needs to become part of your documented upgrade procedure for JunOS SPACE:

  • Change admin user last password change to be today using chage
  • Upgrade SPACE
  • Change admin user expiry to “never” using chage

The Linux command “chage” will show you the current settings for a user and let’s you change those:

  • Log in as admin via ssh
  • Choose “(Debug) run shell”
  • Use “chage” to see and then change the admin password expiry:
    chage -l admin
    Last password change : Mar 15, 2016
    Password expires : May 24, 2016
    Password inactive : never
    Account expires : never
    Minimum number of days between password change : 7
    Maximum number of days between password change : 70
    Number of days of warning before password expires : 7
  • Change the parameters: 

    chage admin
    Changing the aging information for admin
    Enter the new value, or press ENTER for the default

    Minimum Password Age [7]: 0
    Maximum Password Age [70]: 99999
    Last Password Change (YYYY-MM-DD) [2016-03-15]: <today’s date>
    Password Expiration Warning [7]:
    Password Inactive [-1]:
    Account Expiration Date (YYYY-MM-DD) [1969-12-31]:

    And verify:

    chage -l admin
    Last password change : Mar 25, 2016
    Password expires : never
    Password inactive : never
    Account expires : never
    Minimum number of days between password change : 0
    Maximum number of days between password change : 99999
    Number of days of warning before password expires : 7

Reset admin password on JunOS SPACE VM

Picture this: Bob, your JunOS SPACE administrator, left the company. IT diligently wiped his laptop. Bob was the only one who had the admin password (for ssh / CLI access) to your JunOS SPACE installation.

After vowing to do better in future and storing all infrastructure passwords in some form of centralized, encrypted, backed-up password safe, you are left with the task of restoring access.

If you are running your JunOS SPACE instance on an appliance, all you need is a USB stick and Juniper’s instructions. (Note: That may not be entirely true – see discussion of how /etc/shadow behaves in 14.1R2 of SPACE, below)

If you are running your JunOS SPACE instance in a VM, read on.

You’ll need vSphere-level access to the VMWare host JunOS SPACE runs on. Make friends with the VMWare admin.

You’ll need an ISO of a Linux rescue disk.

Power down the SPACE VM and change its boot properties to force boot into BIOS, and delay by 10,000ms for good measure, like so:

Screenshot 2015-07-16 10.44.33

Power on the SPACE VM, and open a Console. You’ll find yourself in the BIOS shortly. Connect the virtual CD drive to the ISO you downloaded (that’s the CD-with-wrench icon in the Console) and change the BIOS to boot from CD first, like so:

Screenshot 2015-07-16 10.54.02

Alternatively, you could have left the BIOS alone and hit ESC to get the boot menu during the 10 second boot delay.

Whichever option you choose, make sure the CD has finished connecting (loading the ISO) before you hit F10/Enter. If you are remote to your VMWare host, it may make sense to upload the ISO to the datastore first, and connect the virtual CD to that copy.

Once the VM has booted from the rescue ISO, I chose the “standard with US keymap” boot option for the rescue disk.

Use lvscan to see the device names of the SPACE disks:

 ACTIVE '/dev/jmpvgnocf/lvroot' [25.41 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvtmp' [17.91 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvvar' [207.78 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvlog' [4.88 GB] inherit

Mount the root partition:

mount /dev/jmpvgnocf/lvroot /mnt/custom

Move the shadow- backup file (don’t delete it):

mv /mnt/custom/etc/shadow- /mnt/custom/root

I tried this without moving shadow- and it would revert to the previous admin password after editing shadow.

Juniper’s original instructions for the appliance, based on the now-ancient SPACE 1.2, want you to save an empty password (::) for admin in shadow. When I tried that, I could no longer log in.

Instead, I used passwd on the rescue CD to get the crypt string for the password “abc123”. It is:

$6$MsjX3IGJ$KVp.m.rQDK5PG1ELcjNajJUbBttXZZzXY/mtQhmNG4PmqSyuVYXgcOjuwjksXkz1rf5DM.laraWKpNf9.T/mp0

With this in hand, I could “vi /mnt/custom/etc/shadow” and insert the string for the new password, which looks like this:

Screenshot 2015-07-16 12.55.28

Save that file using “:x!” in vi and you are ready to “reboot”

Note: Typing that string in the VMWare Console is going to be extremely error-prone. It makes sense to ssh to the rescue Linux instead, so you can copy/paste. Use “ifconfig” to see whether you received a DHCP address. If not, you’ll need to follow the instructions for the rescue image to enable networking – or manually type the password string.

If you changed the BIOS to boot from CDROM first, undo that change when the boot screen comes up.

After reboot, you should be able to log in as admin with the default password and use the option “1” to change the password:

Screenshot 2015-07-16 12.54.59

Now that you’re back in the CLI, you can follow Juniper’s instructions for resetting the password for the ‘super’ user which is used for GUI access and the ‘maintenance’ user which is used for software updates.

With those three accounts restored, you have the access needed to administrate JunOS SPACE again – and set up further GUI authentication for users as desired.

Restoring Extreme Networks Netsight user login

In Extreme Networks’ Netsight management appliance, it is possible to configure external authentication (LDAP or RADIUS) and not set it to “fail to OS,” which is a checkbox that is unchecked by default. If your LDAP or RADIUS server is down, or if you made a mistake entering settings, you’ve just locked yourself out of the Web UI.

There is a way to recover from this without rebuilding Netsight.

I’ll be assuming you still have an OS-level login via ssh to the unit. These instructions assume Netsight on Linux. Netsight on Windows would be similar, you’d just have to figure out where your MySQL utilities live.

This was tested with Netsight 6.3

After logging in to the OS (an ssh session if this is Netsight on Linux), start mysql and connect to the data base:

cd /usr/local/Extreme_Networks/Netsight/mysql/bin
./mysql -unetsight -penterasys --socket /tmp/netsight_mysql.sock -hlocalhost -P4589 netsight

Take a look at the current settings for authentication:

SELECT * FROM nsproperties;

Next, to re-enable Web UI login, you could just set your authentication type back to OS authentication:

UPDATE nsproperties SET VALUE='Default ( OS Authentication )' WHERE NAME='serverAuthType';

Alternatively, you could instruct the authentication to fall back to OS authentication if it fails. You’d have to do this for either LDAP or RADIUS, depending on which one you are using:

UPDATE nsproperties SET VALUE='true' WHERE NAME='serverAuthLDAPFailToOS';
UPDATE nsproperties SET VALUE='true' WHERE NAME='serverAuthRadiusFailToOS';

And for future reference, always check the “Fail To OS” checkbox first before doing any further work in your external authentication settings screen.

Canon EOS utility causes high “System” CPU, network meltdown

This is one of those slightly odd ones. I had high CPU on my PC even when not running anything, 20-25% (one whole core in essence) taken up by “System”. Mouse responsiveness was sluggish, sound would stutter, the works.

I tracked it down to ndis, and my Ethernet driver, using Sysinternals Process Explorer. Other folk have had similar experiences, so I updated the Ethernet driver. Things got worse. Rolled back and disabled all power savings functions on Ethernet. That helped a little, but only a little.

After a few weeks of off-and-on trying to fix this (and business travel in between), it occurred to me that my driver might be behaving just fine, that it was just genuinely busy – that there might be some form of traffic flood on the wire that caused my symptoms.

Sure enough. SSDP (UPnP discovery) packets to an obscene degree, originating from another machine in the network, with the string “Canon” in the discovery.

This was the Canon EOS Utility 2, or rather the UPnP discovery part of it. I saw no way of disabling UPnP discovery in settings, and the user was opposed to uninstalling the utility, so I just renamed the UPnP discovery exe so it couldn’t be started on boot, then rebooted the machine to check that worked. Problem solved. By killing the UPnP discovery process on another PC, my PC is responsive again.

That UPnP discovery process misbehaves. Until Canon fixes the issue – if they will – renaming it is the only workaround I can see.

If your System process CPU is high and it’s your network driver that’s doing it, run wireshark. It might not be your system at fault.

Resolving JunOS Pulse install issue on Windows

On my own Windows 7 machine, I had an odd error when attempting to install JunOS Pulse 5. The installer would start with “copying new files”, then “rolling back action” and would finally tell me that “the wizard was interrupted before JunOS Pulse could be completely installed.”

Usually, when something odd like this happens, the remedy is to uninstall everything Juniper. Which I did. Network Connect, Host Checker, Setup Client, Installer Service, Pulse Collaboration, I got rid of them all. Which didn’t resolve my issue.

Now, this machine has had various versions of Juniper-something installed on it over the years. Even the OAC client at one point. So it was likely that something was lingering that was causing this issue.

I had seen something similar at a customer of mine. They resolved it by re-imagining the machine. I didn’t want to be quite so thorough.

The solution turned out to be to open a command line “as Administrator”, run “pnputil -e > pnplist.txt”, find everything in there that has the name “Juniper” in it, then run “pnputil -d oemXX.inf” for each entry, where “XX” is the actual number of the entry.

This got rid of a lingering driver in the Windows driver store, which was keeping the Pulse driver from being installed. After this little bit of surgery, Pulse installed without complications.

Before reaching for pnputil, be sure that the previous step, uninstalling everything that has to do with Juniper MAG/SA client software, has already been taken. These drivers are components of the client software. Pulling them out from under that client software is not desirable.

I’ve found a forum mention that pnputil helped with a Network Connect install issue where a Juniper driver had gotten orphaned. This remedy is not specific to Pulse alone.