Installing VMWare Tools (Open VM Tools) on JunOS SPACE 16.1 or newer

These instructions are for JunOS SPACE 16.1 or newer. I also have instructions for JunOS SPACE 15.2 or older.

JunOS SPACE, Juniper’s management tool for JunOS devices (switches, routers, firewalls), officially supports Open VM Tools for management from ESXi. Unfortunately, Juniper’s instructions are to build Open VM Tools, and that won’t work without a dev environment, which is not present in JunOS SPACE.

SPACE 16.1 is built on CentOS 6, which in turn is a RHEL 6 clone. Open VM Tools exist in CentOS 6 repositories, so all we need to do is to enable those repositories and we can install binaries.

Install

Navigate to /etc/yum.repos.d and create a new file named centos6.repo, with this content:

[centos6]
name=CentOS 6 Repository
baseurl=http://mirror.centos.org/centos/6/os/$basearch
enabled=1
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/6/os/x86_64/RPM-GPG-KEY-CentOS-6

[extras]
name=CentOS 6 Extras Repository
baseurl=http://mirror.centos.org/centos/6/extras/$basearch
enabled=1
gpgcheck=0

Back on command line, add the EPEL repository:

yum install epel-release

Install Open VM Tools:

yum install open-vm-tools

Start them:

service vmtoolsd start

Cleanup

For good measure, you can now disable the centos 6 and epel repos again, by editing centos6.repo and epel.repo in /etc/yum.repos.d/ and setting this line for centos6, extras, and epel:

enabled=0

Verify those repos are disabled:

yum repolist

 

WSL “Bash on Windows” as a dev environment

I don’t aim to introduce WSL, or do more than link to installation steps.

I do want to take a quick note of tweaks that have been helpful to me in making WSL more useful as a development environment.

  • If you want to use ping: Find the “Bash on Ubuntu on Windows” shortcut, right-click, more, open file location, right-click, properties, advanced, “Run as administrator”. Presto, ping works. This might not be necessary in future builds.
  • Friendlier colors:
    • Edit .bashrc and add to the bottom:
    • LS_COLORS=$LS_COLORS:'di=1;44:' ; export LS_COLORS
    • Edit .vimrc and add:
    • :set background=dark
    • Optional, not entirely certain about this yet: Right-click the bash icon at the upper left of the bash window, choose Defaults, set “Screen Background” to “Black” (0,0,0) and “Screen Text” to “White” (255,255,255)
  • /mnt/* is not a build environment. If you try to compile something you downloaded in Windows from its /mnt/c or /mnt/* location, copy it over to ~/ or /var/tmp first. /mnt/* is not as “Linux-y” a file system as the Ubuntu environment is, and it might (likely: will) trip up your source builds.
  • Coding in perl 5: Works out of the box from Creators Update on. Until then, edit “/usr/lib/perl/5.18.2/Config.pm” and make sure that you have “dont_use_nlink => 1”, around line 94. It defaults to “dont_use_nlink => undef”.
  • Coding in perl 6: Requires a change to the dyncall library to build successfully for now, until dyncall has been updated. The MoarVM github has two alternate patches that will allow it to build, only one is required.
  • Coding in Swift: Similar issue to perl 6, with a patch available. Your best bet is to clear the executable stack flag in libFoundation.so for now, until Swift 3.1 has been released.
  • Coding with a host of other libraries that refuse to link because of execstack, including OpenSSL: Clearing the executable stack flag on the library will work if the library doesn’t require an executable stack. Upstream changes would be best, however, so things start working “out of the box”. Usually the root cause for a library setting the execstack flag is that assembly files are missing a short section to declare the stack not executable. See the Gentoo wiki entry on this. Here’s the code that would go into a .h file included in every .S file, with NO_EXEC_STACK_DIRECTIVE at the end of relevant .S files. NB, this needs to be .S not .s, as a preprocessor is required in order to parse the include file:
  • #if defined(__GNUC__) && defined(__ELF__) && (defined (_linux__) || defined(__FreeBSD__) || defined(__ANDROID__))
    #define NO_EXEC_STACK_DIRECTIVE .section .note.GNU-stack,"",%progbits
    #elsif defined(__SUNPRO_C) && defined(__linux__)
    #define NO_EXEC_STACK_DIRECTIVE .section ".note.GNU-stack"
    #else
    #define NO_EXEC_STACK_DIRECTIVE
    #endif

    That code is portable using GNU as across architectures, and should work with SUNPro Tools aka Oracle Developer Studio.

    Please also see the WSL github discussion regarding execstack. This really can use attention and will have positive impact beyond WSL when resolved. Requesting an executable stack when it’s not needed is an exploit waiting to happen.

JunOS SPACE upgrade hangs at 0%

I attempted to upgrade a JunOS SPACE instance from 15.2R1 to 15.2R2. It would sit at “upgrade process has not started” and 0%. If I changed the URI to the base, I’d be back in the SPACE GUI as if nothing had happened and I had never entered maintenance mode.

This was caused by a failed upgrade months earlier which left a msg.<date> file behind in /var/jmp_upgrade/master/msg . Deleting that file allowed me to successfully upgrade the unit.

After a successful upgrade, the msg/ directory will be empty in both the master and slave directories.

In the process, I learned about a few more files that SPACE looks for. If these exist from a failed upgrade, they can keep a new upgrade from starting. Delete these if they exist:

/var/log/activeUpgradeStatus.log

/var/jmp_upgrade/slave/log/upgradeMetaData.txt

You can find a clue as to why your upgrade is not proceeding in these two directories:

/var/jmp_upgrade/slave/log/

/var/jmp_upgrade/master/log/

Look for log files named after the current and target SPACE version.

Also monitor this file for any issues with maintenance mode:

/tmp/maintenance.log

 

“Cyber Security” for home PCs

Concerns about online security are widespread. No-one wants their logins and finances compromised. How to act on those concerns can be confusing.

How security pros and general users go about securing their devices is quite different. Users often rely on software such as AntiVirus. Security pros likely also use AV, but it’s not their first line of defense.

I’ll share what I consider to be good practice, and what has kept my own machines free from malware for well over two decades now.

  1. Patch religiously, fanatically
  2. Use a password safe and unique passwords
  3. Don’t pirate anything
  4. Be a little paranoid about attachments and links in email
  5. And sure, for defense in depth, run some AV. Chances are it’ll never find anything, though.

If you are only going to do a little, then patch and start using  a password safe. That will give you the biggest bang for your effort.

Let me go into those in some more detail.

Patch religiously, fanatically

This is all about what we security geeks call “attack surface”. The fewer vulnerabilities your system has, the less likely it is to be compromised. The amount of machines that are compromised through known, long-discovered and long-patched vulnerabilities in, say, Adobe Flash, is truly staggering.

So patch religiously. Set everything you can to auto-update. That includes the OS itself, the browser, Java, Flash, Adobe Reader, and really any piece of software that can be updated.

A corollary to this is to reduce the amount of software you need to be on top of.

Not running any Java code? Uninstall Java.

Using a browser that contains its own version of Flash, such as Google Chrome or MS Edge or MS IE 11? Ditch the standalone Flash install.

The main vectors for compromise for a few years running have been Adobe Flash, Adobe Reader, and Oracle Java. Word and Excel get a (dis)honorable mention.

Use a password safe and unique passwords

Passwords are still with us, they’ll continue to be with us for a long time, and they are a terrible way to secure access to important stuff.

So, at the very least, make things easy on yourself and hard on attackers: Use a password safe. There are a number of options available, but if you don’t have very specific criteria, you can’t go far wrong with LastPass. It combines convenience with security.

Convenience is important: If using unique passwords becomes a chore, you likely won’t do it. LastPass will fill in passwords, log you in automatically, generate strong passwords for you and, if you want it to, even change passwords periodically for you.

For your “master password” for LastPass, one good idea is to choose a number of unrelated nouns. An example is “Correct Horse Battery Staple”. Just, for the love of security, do not use that actual example, because it’s a published example. Passwords only work if they are secret.

And then you can start assigning unique, strong passwords to all of your critical accounts. Eventually, all of your accounts. LastPass can help with that chore by running a check on your password database and telling you where you have duplicates and where you have weak passwords.

If you are going to run AntiVirus, there is a copy of LastPass bundled with Webroot, so that’s an option to cut down on the number of software packages you subscribe to.

Don’t pirate anything

What’s this, blogger Dad Mode? The thing about pirated content is that it often comes with something extra, that extra being malware. Once you invite malware into your system, all bets are off. The easiest way to avoid that vector of compromise is to just buy everything outright.

Adult video sites are also notorious for attempts at “drive-by” installs of malware, so browse with care.

Be a little paranoid about attachments and links in email

This is a tough one. Even security pros fall for so-called “spear phishing” attempts, emails with attachments that look legitimate and look like they come from a trusted source, but are actually carriers for malware.

That said, most of those kind of emails are pretty crude. If you’re being asked to “verify your account” or “enter your password here”, that won’t be a legitimate email.  Unless you know you just initiated a password reset yourself and you expect that email. And that’s where it gets a little tough to distinguish between the two. So, be cautious. Check the sender address. When in doubt, manually browse to the site in question, don’t click on the link in the email.

For attachments, if it’s not from a trusted source and you don’t expect it, delete it. No, UPS doesn’t send you word documents. 🙂

Run some AV software

This is really dead-last. AntiVirus software will not detect a lot of malware, and this is meant only to give you one last chance to stop something if all the above defenses fail. If you are not patching religiously and using strong passwords, start there, not here.

I do run AV, as a last-ditch defense if everything else fails, and in the past two decades, my AV hasn’t picked up anything but emails I didn’t act on. I could arguably run without AV and be fine. But then I’d always be wondering whether something slipped through my defenses, after all, so out of an abundance of caution, I pay a subscription for “Medicine”.

Traditional signature-based AntiVirus software can catch maybe 18% of what’s out there, on a good day. So that’s pretty useless. Happily, the industry is evolving.

The best option for a home user – and I say this because as far as I know, it’s the only option for a home user that has modern detection mechanisms – is Webroot, as of November 2016. It happens to come with a copy of Lastpass, reskinned as Webroot Password Manager, so that’s a big plus. Webroot does not do signature-based detection, instead it’s using behavior analysis.

There are other “Next Generation AV” products out there, but nothing else that fits the budget and needs of a home user as far as I am aware.

If you want to add a little bit more protection, then Malwarebytes Anti-Exploit Free is a good choice to protect browsers and Adobe Reader. To get it free, just download the trial and wait for the trial period to expire, then switch it to free mode.

And if you absolutely want more “medicine” and don’t mind paying for it, the full Malwarebytes package is a good choice. I’m running it, but honestly, I wouldn’t install it on my mom’s PC. That’s arguably overkill when patching, secure password use, Webroot, and Anti-Exploit Free are already in place.

 

Convert Windows boot from BIOS to UEFI without decrypting Bitlocker

I wanted to convert an MBR/BIOS boot drive to GPT/UEFI, but without needing to decrypt and then re-encrypt Bitlocker. Mainly because I am lazy. This worked, but I’ll warn that the advice to decrypt completely first is without any doubt the safest way to go.

Follow the instructions for converting from BIOS to UEFI boot.

With the following changes:

  • Print out your Bitlocker Recovery Key from Control Panel, Bitlocker. You will need this key.
  • Take a backup. No seriously. Use Veeam Endpoint if you don’t have anything else installed to take backups. Stuff goes wrong with computers, and you don’t want to lose your system installation and data.
  • Suspend Bitlocker protection on your system drive.
  • Reboot from your Windows installation / recovery DVD/USB, verify that you can get to your c:\windows directory. This might be d:\windows if you have a recovery partition at the end of the disk.
  • Now boot into Windows and convert to GPT using gptgen
  • When you then boot into the Windows installation / recovery media, you’ll be asked for the Bitlocker Recovery Key. After that, the rest of the steps are as in the generic instructions.
  • When booting into Windows (assuming you changed your BIOS to boot (U)EFI instead of MBR now), you’ll be asked for the recovery key again.
  • In my case, the drive didn’t show suspended. I suspended it again.
  • After that, resuming Bitlocker encryption will fail with “The System cannot find the file specified”.
  • Open Explorer, navigate to C:\Windows\System32\Recovery and rename the file “ReAgent.xml” to “ReAgent.xml.old”
  • Resume Bitlocker encryption on drive C:\ This should now succeed.
  • Reboot for good measure to verify that everything works and you don’t get prompted for the recovery key any more.

Change admin user password expiry on JunOS SPACE

The admin user on JunOS SPACE, which is used for ssh / root access, has a default password expiry of 70 days. This may not be desired.

NB: An upgrade of the JunOS SPACE platform will set the admin password expiry to the default of 70 days again. To avoid the admin user password being prompted for change after the upgrade, this procedure needs to become part of your documented upgrade procedure for JunOS SPACE:

  • Change admin user last password change to be today using chage
  • Upgrade SPACE
  • Change admin user expiry to “never” using chage

The Linux command “chage” will show you the current settings for a user and let’s you change those:

  • Log in as admin via ssh
  • Choose “(Debug) run shell”
  • Use “chage” to see and then change the admin password expiry:
    chage -l admin
    Last password change : Mar 15, 2016
    Password expires : May 24, 2016
    Password inactive : never
    Account expires : never
    Minimum number of days between password change : 7
    Maximum number of days between password change : 70
    Number of days of warning before password expires : 7
  • Change the parameters: 

    chage admin
    Changing the aging information for admin
    Enter the new value, or press ENTER for the default

    Minimum Password Age [7]: 0
    Maximum Password Age [70]: 99999
    Last Password Change (YYYY-MM-DD) [2016-03-15]: <today’s date>
    Password Expiration Warning [7]:
    Password Inactive [-1]:
    Account Expiration Date (YYYY-MM-DD) [1969-12-31]:

    And verify:

    chage -l admin
    Last password change : Mar 25, 2016
    Password expires : never
    Password inactive : never
    Account expires : never
    Minimum number of days between password change : 0
    Maximum number of days between password change : 99999
    Number of days of warning before password expires : 7

Reset admin password on JunOS SPACE VM

Picture this: Bob, your JunOS SPACE administrator, left the company. IT diligently wiped his laptop. Bob was the only one who had the admin password (for ssh / CLI access) to your JunOS SPACE installation.

After vowing to do better in future and storing all infrastructure passwords in some form of centralized, encrypted, backed-up password safe, you are left with the task of restoring access.

If you are running your JunOS SPACE instance on an appliance, all you need is a USB stick and Juniper’s instructions. (Note: That may not be entirely true – see discussion of how /etc/shadow behaves in 14.1R2 of SPACE, below)

If you are running your JunOS SPACE instance in a VM, read on.

You’ll need vSphere-level access to the VMWare host JunOS SPACE runs on. Make friends with the VMWare admin.

You’ll need an ISO of a Linux rescue disk.

Power down the SPACE VM and change its boot properties to force boot into BIOS, and delay by 10,000ms for good measure, like so:

Screenshot 2015-07-16 10.44.33

Power on the SPACE VM, and open a Console. You’ll find yourself in the BIOS shortly. Connect the virtual CD drive to the ISO you downloaded (that’s the CD-with-wrench icon in the Console) and change the BIOS to boot from CD first, like so:

Screenshot 2015-07-16 10.54.02

Alternatively, you could have left the BIOS alone and hit ESC to get the boot menu during the 10 second boot delay.

Whichever option you choose, make sure the CD has finished connecting (loading the ISO) before you hit F10/Enter. If you are remote to your VMWare host, it may make sense to upload the ISO to the datastore first, and connect the virtual CD to that copy.

Once the VM has booted from the rescue ISO, I chose the “standard with US keymap” boot option for the rescue disk.

Use lvscan to see the device names of the SPACE disks:

 ACTIVE '/dev/jmpvgnocf/lvroot' [25.41 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvtmp' [17.91 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvvar' [207.78 GB] inherit
 ACTIVE '/dev/jmpvgnocf/lvlog' [4.88 GB] inherit

Mount the root partition:

mount /dev/jmpvgnocf/lvroot /mnt/custom

Move the shadow- backup file (don’t delete it):

mv /mnt/custom/etc/shadow- /mnt/custom/root

I tried this without moving shadow- and it would revert to the previous admin password after editing shadow.

Juniper’s original instructions for the appliance, based on the now-ancient SPACE 1.2, want you to save an empty password (::) for admin in shadow. When I tried that, I could no longer log in.

Instead, I used passwd on the rescue CD to get the crypt string for the password “abc123”. It is:

$6$MsjX3IGJ$KVp.m.rQDK5PG1ELcjNajJUbBttXZZzXY/mtQhmNG4PmqSyuVYXgcOjuwjksXkz1rf5DM.laraWKpNf9.T/mp0

With this in hand, I could “vi /mnt/custom/etc/shadow” and insert the string for the new password, which looks like this:

Screenshot 2015-07-16 12.55.28

Save that file using “:x!” in vi and you are ready to “reboot”

Note: Typing that string in the VMWare Console is going to be extremely error-prone. It makes sense to ssh to the rescue Linux instead, so you can copy/paste. Use “ifconfig” to see whether you received a DHCP address. If not, you’ll need to follow the instructions for the rescue image to enable networking – or manually type the password string.

If you changed the BIOS to boot from CDROM first, undo that change when the boot screen comes up.

After reboot, you should be able to log in as admin with the default password and use the option “1” to change the password:

Screenshot 2015-07-16 12.54.59

Now that you’re back in the CLI, you can follow Juniper’s instructions for resetting the password for the ‘super’ user which is used for GUI access and the ‘maintenance’ user which is used for software updates.

With those three accounts restored, you have the access needed to administrate JunOS SPACE again – and set up further GUI authentication for users as desired.