Reset admin password on JunOS SPACE VM

Picture this: Bob, your JunOS SPACE administrator, left the company. IT diligently wiped his laptop. Bob was the only one who had the admin password (for ssh / CLI access) to your JunOS SPACE installation.

After vowing to do better in future and storing all infrastructure passwords in some form of centralized, encrypted, backed-up password safe, you are left with the task of restoring access.

If you are running your JunOS SPACE instance on an appliance, all you need is a USB stick and Juniper’s instructions. (Note: That may not be entirely true – see discussion of how /etc/shadow behaves in 14.1R2 of SPACE, below)

If you are running your JunOS SPACE instance in a VM, read on.

You’ll need vSphere-level access to the VMWare host JunOS SPACE runs on. Make friends with the VMWare admin.

You’ll need an ISO of a Linux rescue disk.

Power down the SPACE VM and change its boot properties to force boot into BIOS, and delay by 10,000ms for good measure, like so:

Screenshot 2015-07-16 10.44.33

Power on the SPACE VM, and open a Console. You’ll find yourself in the BIOS shortly. Connect the virtual CD drive to the ISO you downloaded (that’s the CD-with-wrench icon in the Console) and change the BIOS to boot from CD first, like so:

Screenshot 2015-07-16 10.54.02

Alternatively, you could have left the BIOS alone and hit ESC to get the boot menu during the 10 second boot delay.

Whichever option you choose, make sure the CD has finished connecting (loading the ISO) before you hit F10/Enter. If you are remote to your VMWare host, it may make sense to upload the ISO to the datastore first, and connect the virtual CD to that copy.

Once the VM has booted from the rescue ISO, I chose the “standard with US keymap” boot option for the rescue disk.

Use lvscan to see the device names of the SPACE disks:

 ACTIVE '/dev/jmpvgnocf/lvroot' [25.41 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvtmp' [17.91 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvvar' [207.78 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvlog' [4.88 GB] inherit

Mount the root partition:

mount /dev/jmpvgnocf/lvroot /mnt/custom

Move the shadow- backup file (don’t delete it):

mv /mnt/custom/etc/shadow- /mnt/custom/root

I tried this without moving shadow- and it would revert to the previous admin password after editing shadow.

Juniper’s original instructions for the appliance, based on the now-ancient SPACE 1.2, want you to save an empty password (::) for admin in shadow. When I tried that, I could no longer log in.

Instead, I used passwd on the rescue CD to get the crypt string for the password “abc123″. It is:

$6$MsjX3IGJ$KVp.m.rQDK5PG1ELcjNajJUbBttXZZzXY/mtQhmNG4PmqSyuVYXgcOjuwjksXkz1rf5DM.laraWKpNf9.T/mp0

With this in hand, I could “vi /mnt/custom/etc/shadow” and insert the string for the new password, which looks like this:

Screenshot 2015-07-16 12.55.28

Save that file using “:x!” in vi and you are ready to “reboot”

Note: Typing that string in the VMWare Console is going to be extremely error-prone. It makes sense to ssh to the rescue Linux instead, so you can copy/paste. Use “ifconfig” to see whether you received a DHCP address. If not, you’ll need to follow the instructions for the rescue image to enable networking – or manually type the password string.

If you changed the BIOS to boot from CDROM first, undo that change when the boot screen comes up.

After reboot, you should be able to log in as admin with the default password and use the option “1” to change the password:

Screenshot 2015-07-16 12.54.59

Now that you’re back in the CLI, you can follow Juniper’s instructions for resetting the password for the ‘super’ user which is used for GUI access and the ‘maintenance’ user which is used for software updates.

With those three accounts restored, you have the access needed to administrate JunOS SPACE again – and set up further GUI authentication for users as desired.

Restoring Extreme Networks Netsight user login

In Extreme Networks’ Netsight management appliance, it is possible to configure external authentication (LDAP or RADIUS) and not set it to “fail to OS,” which is a checkbox that is unchecked by default. If your LDAP or RADIUS server is down, or if you made a mistake entering settings, you’ve just locked yourself out of the Web UI.

There is a way to recover from this without rebuilding Netsight.

I’ll be assuming you still have an OS-level login via ssh to the unit. These instructions assume Netsight on Linux. Netsight on Windows would be similar, you’d just have to figure out where your MySQL utilities live.

This was tested with Netsight 6.3

After logging in to the OS (an ssh session if this is Netsight on Linux), start mysql and connect to the data base:

cd /usr/local/Extreme_Networks/Netsight/mysql/bin
./mysql -unetsight -penterasys --socket /tmp/netsight_mysql.sock -hlocalhost -P4589 netsight

Take a look at the current settings for authentication:

SELECT * FROM nsproperties;

Next, to re-enable Web UI login, you could just set your authentication type back to OS authentication:

UPDATE nsproperties SET VALUE='Default ( OS Authentication )' WHERE NAME='serverAuthType';

Alternatively, you could instruct the authentication to fall back to OS authentication if it fails. You’d have to do this for either LDAP or RADIUS, depending on which one you are using:

UPDATE nsproperties SET VALUE='true' WHERE NAME='serverAuthLDAPFailToOS';
UPDATE nsproperties SET VALUE='true' WHERE NAME='serverAuthRadiusFailToOS';

And for future reference, always check the “Fail To OS” checkbox first before doing any further work in your external authentication settings screen.

Better Lync / Skype meeting locations for mobile users

By default, the “Location” for a Skype for Business (ne Lync) Meeting in an Outlook calender invite reads as “Skype Meeting”. This is not very friendly to mobile users who want to dial in via phone. If the body of the meeting contains a lot of agenda text, the Skype dialin information may not display at all on mobile; and without something in the “Location”, a user can’t just tap the invite to dial in directly.

I wanted to have a Macro that lets me set the “Location” to “phone-number x conference-id#”. This way, the user can dial, and has the conference id available in the dial screen to be sent with one touch.

Further, I wanted to be able to control permissions through the Skype Meeting Options, which means I can’t use a “dedicated meeting space”, but have to use the “new meeting space” option, which means the conference ID changes with every meeting.

I want to hear when people enter or leave, I don’t want anyone to have to wait around in a lobby, and I conference with customers, so I don’t want restrictions as to who can join or present. The screenshot below shows the permissions I chose before hitting “Remember Settings”.

Screenshot 2015-06-01 10.33.50

The macro I created is not very smart: The phone number to call is hard-coded, which means you will need to enter your number into the code. If your phone number is not static, then adapting the code to regex the phone number is left as an exercise to you.

In order to use the macro, you’ll need to:

– Enable access to the coding tools in Outlook, the “Developer” toolbar

– Create the macro

– Sign the macro and save

– Link the macro to the “New Appointment” screen

The macro was created for use with Outlook 2010. I expect it will work with newer versions such a 2013 and 2016 as well, but this has not been tested.

Enable access to the coding tools in Outlook

From the Outlook main windows, click on “File” then “Options”

In the “Outlook Options” window, click on “Customize Ribbon” on the left. Check the “Developer” ribbon to show up.

Click OK

Screenshot 2015-06-01 10.41.13

Create the macro

From the Outlook main window, click the “Developer” toolbar, then the “Visual Basic” icon

Right-click the Project name in the left pane, choose “Properties…” and set the “Project Name” to “Skype”, then click OK.

Under “Modules”, you should see a “Module 1″ with an empty window to the right. Click it and paste the code below.

<Edit> Now with code formatting that will paste correctly.

Sub AddLocation()
Application.ActiveInspector.CurrentItem.Location = "(xxx)-yyy-zzzz x " & GetValueUsingRegEx() & "#"
End Sub

Function GetValueUsingRegEx() As String
 ' Set reference to VB Script library
 ' Microsoft VBScript Regular Expressions 5.5
 
 Dim olAppt As Outlook.AppointmentItem
 Dim Reg1 As RegExp
 Dim M1 As MatchCollection
 Dim M As Match
 
 Set olAppt = Application.ActiveInspector.CurrentItem
 ' Debug.Print olAppt.Body
 
 Set Reg1 = New RegExp
 
 ' \s* = invisible spaces
 ' \d* = match digits
 ' \w* = match alphanumeric
 
 With Reg1
 .Pattern = "Conference ID\s*[:]+\s*(\d*)\s*"
 .Global = True
 End With
 If Reg1.test(olAppt.Body) Then
 
 Set M1 = Reg1.Execute(olAppt.Body)
 Set M = M1(0)
 ' Debug.Print M
 ' Debug.Print M.SubMatches(0)
 GetValueUsingRegEx = M.SubMatches(0)

 End If
 
End Function

In that code, change (xxx)-yyy-zzzz in the second line to the actual phone number you want to show up in the “Location” of your meeting.

In the Visual Basic editor, click the “Tools” menu, then “References…” and check the “Microsoft VBScript Regular Expressions 5.5″, then click “OK”. This is required for the macro to function.

Screenshot 2015-06-01 11.11.32

Here is a screen shot of the VBA editor for reference:

Screenshot 2015-06-01 10.44.26

Sign the macro and save

Outlook 2010, by default, has a macro security setting of “Notifications for digitally signed macros, all other macros disabled”. Without signing the macro, it may work the first time around, but you might get an error message that “the macros in this project are disabled” in future.

We’ll create a self-signed cert and apply it. For Outlook 2010 and 2013, look for the “Digital Certificate for VBA Projects” application in the Start menu, give it your name, and click “OK” to create the certificate, like so:

Screenshot 2015-06-02 13.47.47

Next, apply this certificate to your project. In the VBA editor, choose the “Tools” menu then “Digital Signature…” and apply the certificate you just created by selecting it via “Choose…” then clicking “OK”:

Screenshot 2015-06-02 13.48.10

 

Lastly, save your Project via “File” and “Save VbaProject.OTM”.

When that is done, you can close the VBA editor screen either via the X in the corner or through the “File” menu.

At some point when first executing the macro, you may see a warning whether to trust the certificate you just created. Choose “Enable Macros” or “Trust all documents from this publisher” :

Screenshot 2015-06-02 13.53.19

 

Link the macro to the “New Appointment” screen

This was actually the least intuitive step of the whole lot.

In Outlook, click on “Calendar”, then “New Skype Meeting”. As far as I can tell, you can only add the macro to the ribbon while in that window.

With that new meeting window open, click “File”, then “Options”.

Click on “Customize Ribbon”

Right-click “Appointment” and choose “Add New Group”

Right-Click the “New Group (Custom)” and rename it to “Skype”

Select “Skype (Custom)” and use the up-arrow to the right to move it just under “Skype Meeting”

With “Skype (Custom)” still selected, change “Choose commands from:” on the left-hand list to “Macros”

Click on the “Skype.AddLocation” macro and use the “Add>>” button

The macro should now show up in your “Skype (Custom)” group.

Click OK and test the macro!

Screenshot of what this ribbon option window looks like below.

Screenshot 2015-06-01 10.52.39

All Done

This is a fair serious amount of effort just to get a phone number into the “Location” field. For me, it was worth it because that effort makes life easier for my customers joining my meetings.

If so desired, you can disable the “Developer” toolbar again.

Canon EOS utility causes high “System” CPU, network meltdown

This is one of those slightly odd ones. I had high CPU on my PC even when not running anything, 20-25% (one whole core in essence) taken up by “System”. Mouse responsiveness was sluggish, sound would stutter, the works.

I tracked it down to ndis, and my Ethernet driver, using Sysinternals Process Explorer. Other folk have had similar experiences, so I updated the Ethernet driver. Things got worse. Rolled back and disabled all power savings functions on Ethernet. That helped a little, but only a little.

After a few weeks of off-and-on trying to fix this (and business travel in between), it occurred to me that my driver might be behaving just fine, that it was just genuinely busy – that there might be some form of traffic flood on the wire that caused my symptoms.

Sure enough. SSDP (UPnP discovery) packets to an obscene degree, originating from another machine in the network, with the string “Canon” in the discovery.

This was the Canon EOS Utility 2, or rather the UPnP discovery part of it. I saw no way of disabling UPnP discovery in settings, and the user was opposed to uninstalling the utility, so I just renamed the UPnP discovery exe so it couldn’t be started on boot, then rebooted the machine to check that worked. Problem solved. By killing the UPnP discovery process on another PC, my PC is responsive again.

That UPnP discovery process misbehaves. Until Canon fixes the issue – if they will – renaming it is the only workaround I can see.

If your System process CPU is high and it’s your network driver that’s doing it, run wireshark. It might not be your system at fault.

Installing VMWare Tools on JunOS SPACE

JunOS SPACE, Juniper’s management platform for JunOS devices (switches, routers, firewalls) does not come with gcc or kernel-headers. Installing VMWare Tools from a mounted ISO via vmware-install.pl is not all that successful. Happily, VMWare still provides RPM versions of those tools. SPACE 13.3 is built on CentOS 6, which in turn is a RHEL 6 clone.

This post will be around when SPACE 13.3 is history. The easiest One way to figure out what version of CentOS it might, somewhat loosely, be based on, is to run uname -a and take a look at the kernel version:

uname -a

Linux space-0050568776bc 2.6.32-100.24.1.el5 #1 SMP Tue Aug 20 12:17:49 CST 2013 x86_64 x86_64 x86_64 GNU/Linux

Cross-reference to the Wiki entry for CentOS and you can see that 2.6.32-xx is some version of CentOS 6.

Note: Check the comments below, not all versions of SPACE are created equal. Yours might be based on RHEL 5/CentOS 5. There’s a faster way to check for version, suggested by commenter Riley: “rpm -qa | grep el”. Be sure to use the VMWare Tools package that matches the version of CentOS your SPACE is running.

1) Start by downloading the RPMs for VMWare Tools on RHEL 6. You’ll want the following (or their current equivalent):

vmware-tools-core-9.4.5-1.el6.x86_64.rpm
vmware-tools-esx-nox-9.4.5-1.el6.x86_64.rpm
vmware-tools-foundation-9.4.5-1.el6.x86_64.rpm
vmware-tools-guestlib-9.4.5-1.el6.x86_64.rpm
vmware-tools-libraries-nox-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-autoUpgrade-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-deployPkg-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-grabbitmqProxy-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-guestInfo-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-hgfsServer-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-powerOps-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-timeSync-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-vix-9.4.5-1.el6.x86_64.rpm
vmware-tools-plugins-vmbackup-9.4.5-1.el6.x86_64.rpm
vmware-tools-services-9.4.5-1.el6.x86_64.rpm

That list might change with newer versions of the tools and of RHEL, of course. When in doubt, grab just the vmware-tools-esx-nox package, try to install it, and take a note of all the dependent packages it wants, then download those too.

 

2) scp the lot to SPACE, say to /var/tmp. While WinSCP is unhappy with the shell the admin user runs on, command-line scp does not care and will work. Choose any version you like: The one that comes with Putty, the one that comes with Cygwin, or any other. And if you’re running OSX or Linux, you can feel extra-smug because you have scp as part of your base OS.

 

3) Install those RPMs. Now, you could install the GPG key they are signed with, but if you trust that you got them from VMWare, in an unaltered form, then just:

yum install --nogpgcheck vmware-tools*rpm

You’ll notice some errors about initctl missing, and that means the service didn’t get installed. We’ll fix that next.

 

4) Copy the vmware-tools-services script over to init.d, and rename it while you’re at it

cp /etc/vmware-tools/init/vmware-tools-services /etc/init.d/vmware-tools

 

5) Edit /etc/init.d/vmware-tools with vi and add two lines, so chkconfig knows what to do with it:

# chkconfig: 345 20 80
# description: VMWare Tools

6) And add it to startup:

chkconfig --add vmware-tools

If you like, you can use

chkconfig --list

to make sure that worked.

 

7) Last, start the tools

service vmware-tools start

And satisfy yourself that this worked, too:

ps -ef | grep vmtoolsd

 

vSphere should now be reporting that SPACE is running “VMtools 3rd party/independent”. And that’s all there is to it.

The kmod portion of the tools won’t install, by the way – but then it’s not needed.

 

Resolving JunOS Pulse install issue on Windows

On my own Windows 7 machine, I had an odd error when attempting to install JunOS Pulse 5. The installer would start with “copying new files”, then “rolling back action” and would finally tell me that “the wizard was interrupted before JunOS Pulse could be completely installed.”

Usually, when something odd like this happens, the remedy is to uninstall everything Juniper. Which I did. Network Connect, Host Checker, Setup Client, Installer Service, Pulse Collaboration, I got rid of them all. Which didn’t resolve my issue.

Now, this machine has had various versions of Juniper-something installed on it over the years. Even the OAC client at one point. So it was likely that something was lingering that was causing this issue.

I had seen something similar at a customer of mine. They resolved it by re-imagining the machine. I didn’t want to be quite so thorough.

The solution turned out to be to open a command line “as Administrator”, run “pnputil -e > pnplist.txt”, find everything in there that has the name “Juniper” in it, then run “pnputil -d oemXX.inf” for each entry, where “XX” is the actual number of the entry.

This got rid of a lingering driver in the Windows driver store, which was keeping the Pulse driver from being installed. After this little bit of surgery, Pulse installed without complications.

Before reaching for pnputil, be sure that the previous step, uninstalling everything that has to do with Juniper MAG/SA client software, has already been taken. These drivers are components of the client software. Pulling them out from under that client software is not desirable.

I’ve found a forum mention that pnputil helped with a Network Connect install issue where a Juniper driver had gotten orphaned. This remedy is not specific to Pulse alone.

 

Restoring a bitlocker system volume with Acronis 2014

I had reason to restore my Windows 8 system volume, which is encrypted using Bitlocker. Getting access to my data drive back after that wasn’t quite as straightforward as I had hoped. For reference, and in case others get into this situation, here is what I encountered.

My setup

One system drive, SSD, encrypted using Bitlocker, running Windows 8.1.

One data drive, HDD, encrypted using Bitlocker, set to auto-unlock.

One backup drive, HDD, unencrypted. TrueImage 2014 is set to encrypt the backups themselves. This is crucial: Without an unencrypted backup drive, I couldn’t “get at” my backups when restoring the system drive.

I do not have a TPM module and use a USB stick instead for Bitlocker keys. I do not have Secure Boot enabled, mainly because I upgraded this system from Windows 7, don’t have a Secure Boot compatible GPU, and really don’t feel like re-installing Windows 8 to get the additional boot sector protection of Secure Boot. It’s a neat feature, but convenience wins out.

I keep a copy of my startup key and my recovery keys on a separate USB stick in the safe, and this proved to have been a necessary precaution.

Restore system drive

Restoring the system drive itself was reasonably straightforward. It cannot be done from Acronis within Windows, I had to use a Rescue CD instead. On machines without an optical disk drive, use a Rescue USB stick.

As expected, the system drive was unencrypted after the restore. This is a result of the way Acronis takes sector backups: It is “fed” the unencrypted data by Bitlocker, and so that is what get’s backed up and restored.

Get access to data drive back

I encountered two errors.

First, upon attempting to unlock my data drive, I received an error “Application not found”. The context menu entry to unlock the drive read “unlock-bde”, which points to an issue. This can be resolved by editing the registry, see Microsoft’s KB entry. The automatic fixit didn’t work for me, since my temp directory is on the data drive. Rather than change the location of temp, I just made the necessary two changes in regedit. To unlock the drive, I had to get my recovery key USB stick from the safe. You do have one of those, I’d hope. If not, you might be screwed.

Secondly, upon attempting to set the data drive to auto-unlock, I received an error “data error cyclic redundancy check”. No need to panic, the data is fine: This is a problem with the stored Bitlocker keys. Mark Berry documented the fix back in 2010. I used his updated (2/17/2011) methodology, which is henceforth no longer untested. In a nutshell, enable Bitlocker on the system drive, reboot. While the system drive is encrypting, use manage-bde to get rid of old auto-unlock keys and delete external keys from data volumes, then re-enable auto-unlock. This worked like a charm. Note he uses S: as a sample drive letter of the data volume; replace with whatever drive letter your data volume has.

Lastly, do not forget to copy your startup key and backup your new recovery key for the system volume onto your “oh crap” USB stick, and put it back in the safe where it belongs.