Fortigate performance – gotchas in hardware acceleration

Fortinet’s Fortigate firewalls have amazing performance for the dollar, all thanks to their strength in ASIC design: They use custom hardware chips to accelerate everything from straight packet-slinging to encryption to content inspection.

When that hardware acceleration switches off, you may find yourself with terrible performance and CPU spikes. This article aims to document some of the “gotchas” I am aware of.

Step one: Docs. These are your friend. The hardware acceleration guide and the Cookbook entry on accelerated content inspection are good ones to bookmark. They’ll be your “source of truth” unless your hardware shows you otherwise.

Gotcha: Crypto Primitives

Which “cryptographic primitives” are accelerated depends on the generation of Fortigate firewall you are using. For example, the “C” series does not accelerate SHA-256, but does accelerate SHA-1. Having all your IPSEC tunnels hit CPU hurts.

You are in the clear on “D” series and above, which means as of 2019, you’re not terribly likely to encounter this issue with the most popular primitives. Still, before deciding on a standard for your VPNs, check the acceleration guide.

Gotcha: Inter-VDOM links

We had FTP transfers that traversed VDOMs completely kill our firewall, and we couldn’t make sense of it. It turns out, inter-VDOM links on NP4 and earlier processors aren’t accelerated at all. On NP6, they can be accelerated, if you took the trouble to configure that. Two accelerated inter-VDOM links with two interfaces each are available. The acceleration guide explains how to configure that on NP6.

If you are on hardware that does not accelerate inter-VDOM links, or you’re out of accelerated links, you do have the option of using accelerated hardware ports instead. Assign a (pair of) ports to each VDOM, and cable them together. Presto, accelerated inter-VDOM link that actually flows over physical interfaces. It’s not pretty, but sometimes it’s the only solution.

Gotcha: Proxy Mode

This one is simple. In proxy mode, content inspection isn’t accelerated. Use flow mode instead. This does put a bit of a crimp in the “our inspection is superior because we can proxy” messaging by Fortinet.

Since I had this statement challenged, here is the documentation I use to come to that conclusion.

“Firewall sessions that include proxy-based security profiles or a mixture of flow-based and proxy-based security profiles are never offloaded to network processors and are always processed by the FortiGate CPU.”, stated in the offloading Cookbook guide.

The same language is in the acceleration guide: “Firewall sessions that include proxy-based security profiles are never offloaded to network processors and are always processed by the FortiGate CPU”

Gotcha: Softswitch

Any traffic traversing a softswitch won’t be accelerated. The only environment where this is likely to matter is a SoHo setup with one of the smaller SoC-based Fortigates, anything below a FG-100. In a SoHo environment, it can be desirable to have WiFi and LAN on the same subnet, so that home systems can detect each other. A softswitch offers an easy solution, and also happens to switch off acceleration entirely.

Even with acceleration off, the performance of a SoHo Fortigate is often more than sufficient for what little traffic that environment generates.

If you’d like to use acceleration, simply “dump” the WiFi traffic onto the wire, that is, configure the SSID for the home network in bridge mode, not tunnel mode. You can even configure that bridged traffic to use VLAN tagging, if you’d like to keep the CAPWAP AP subnet and the home subnet separate.

And use the hardware switch built into the SoHo Fortigates for the built-in LAN ports. Without a softswitch, traffic will be accelerated. In the GUI, you can see your nTurbo and SPU sessions: If they’re 0, you’re not accelerating anything.

As this is configured per SSID, you can still have a second SSID in tunnel mode for IoT and guest traffic. In my own home, IoT devices and guests are on a tunneled SSID that only has access to the Internet; and WiFi printers are on a bridged SSID that shares a subnet with the wired LAN, making it easy to discover these printers from the PCs in the household.


Does it make sense to keep your emergency fund in a HELOC?

For a change, a personal finance post, not a technical one.

There’s this idea that you can use a Home Equity Line Of Credit (HELOC) as your emergency fund, and pocket (or invest) the difference between the interest rate on your mortgage and the interest rate of wherever your emergency fund was sitting. It takes a lot of discipline and is definitely not worth it if you are going to tap the HELOC just because the money is there, not because you have a genuine emergency – and is only worth it if tapping your emergency fund is unlikely, because you have stable employment.

MoneyMetaGame walks you through some of the considerations, and DoughRoller gives you some additional factors to consider.

I did some quick back-of-the-napkin math, and in my case, replacing my Sallie Mae money market account with a HELOC is not a good idea. That’s simply because of where I find myself in the interest cycle.

I got my mortgage at 2.875%. Sallie Mae currently pays me 2%. My HELOC is considerably above that, in the 6-ish range. So, given an emergency fund of say $20k, I could save myself $175/year if I applied that to principal and relied on the HELOC for emergencies instead. It’s a relatively low amount because I got my mortgage when rates where near the bottom, and rates are now climbing up again.

In an emergency, I’d pay $625/year ((HELOC rate minus mortgage rate) times 20k) for dipping into the HELOC. Keeping in mind this may be higher: The HELOC uses a variable rate tied to prime. Now, hopefully, I wouldn’t need the emergency money for a full year. And, still, with the cost of using the HELOC, which is higher the higher the prime rate goes, the marginal utility of an extra $175/year, the fact that my money market account may eventually outperform my mortgage if prime keeps rising, and the fact that the HELOC might get closed on me in a downturn, I’ve decided I’m better off just leaving my emergency fund in a Money Market account. Sallie Mae pays amazingly good interest on those.

How to move from OneNote 2016 to OneNote Windows 10

Microsoft have stated that Windows 10 OneNote, the UWP version that installs from the store, is going to be the only one receiving new features. And OneNote 2016, the desktop client, is going to stay as-is.

If, like me, you have a number of OneNote 2016 Notebooks, you may be stumped as to how to transfer these for use in (UWP) OneNote. Here’s how.

  • Open OneNote 2016 (should also work with 2013, 2010)
  • Right-click the Notebook you’d like to move over, choose to “Share This Notebook”, choose “OneDrive” (in my case “OneDrive – Personal as this is not a corporate account), give it a “Notebook Name” and tell OneNote 2016 to “Move Notebook” to your OneDrive folder. Be sure to be signed in with the same account you’ll be using in OneNote. You’ll get a notice that your Notebook is now syncing to the new location. It should automatically be shared with your MS account as the “owner”, which suffices. Use the back arrow to leave this screen.
  • Right-click the Notebook again and choose “Notebook Sync Status…”, wait until the Notebook is fully synchronized.
  • Go to and open the newly shared Notebook by clicking on its name.
  • At this point, back in OneNote, you can choose “More Notebooks…” and your Notebook will show up. Open it.

Without the step of opening the Notebook in the online version of OneNote, this did not work for me. It’d be shared but never show up in OneNote UWP.

Also, “export” does nothing for you here: OneNote UWP as of July 2018 cannot open files, neither .onepgk nor .mht.

IPv6 with Prefix Delegation on Fortigate

[Edit 2018-10-18] Make sure to use FortiOS 6.0.3 or later for this, as earlier versions of 6.0.x will force your interface to IPv6 “static” when you make any change to the interface from the GUI, including changes to its IPv4 configuration, such as a DHCP reservation. I have not tested 5.6.x but am assuming it has the same issue.

This post is meant to be a full description of how to enable IPv6 connectivity on an ISP link with Prefix Delegation, using a Fortigate firewall. I’ll use Comcast as an example, since that’s my ISP.

This post focuses on home / home office connections, though a small business that uses the Fortigate unit as the LAN router would work the same way. If you use an ISP link with Prefix Delegation but have an internal core router downstream from the Fortigate, you may need a static IPv6 prefix instead.

I am not covering how link failover / SDWAN would work with IPv6. It’s an interesting use case, and I lack the second link to test it.

There are three components to setting up IPv6 in this environment.

  • Receiving an external IP and a prefix using Prefix Delegation
  • Assigning subnets to Fortigate internal interfaces and assigning addresses to client devices
  • IPv6 firewall policy

This post pulls together information already available elsewhere. I have given references at the end of the post.

A quick IPv6 refresher

There are just a few things to remember for home / office use if you are coming from an IPv4 world. This post does not apply to Enterprise networks, though I mention Enterprise for reference here and there.

– Your “site” (home, office) will receive a /64 or /60 prefix from Comcast (residential), or as large as /56 (business). (A /48 is the typical Enterprise site prefix size.)

– All local networks (subnets) have a /64 prefix length. Subnetting further really isn’t a thing, with the exception of /127 point to point links, done for security reasons. You can have more than one /64 on one VLAN and clients can have more than one IPv6 address.

– There is no NAT. All your clients will have public addresses. (This might not be true in Enterprise networks where you may decide to either use public addresses or ULAs with NPT, Network Prefix Translation.)

– ICMPv6 is crucial to connection health. Just dropping all ICMP at the border won’t do the trick.

– In an IPv6 address, the first four fields are the network, the last four fields the device. Leading zeroes can be omitted, and a bunch of zeroes can be summarized as :: – with the caveat that there’s only one :: per address. For example, 2001:db8:3c4d:f40::/64 might be your subnet, and 2001:db8:3c4d:f40::1/64 is the address assigned to your Fortigate interface on that subnet.

– DHCPv6 cannot assign a next hop. Not all client operating systems can receive a DNS server without DHCPv6.

Receiving a prefix via Prefix Delegation

Before you get started, make sure that IPv6 is turned on in “System -> Feature Visibility”.

On a residential or business line, your ISP will assign you a prefix to use for your internal network(s). This prefix is received on your ISP-facing interface via DHCPv6 Prefix Delegation (PD), and can then be assigned dynamically to your internal interface(s).

Comcast will assign you a delegated /64 or /60 prefix on a residential line. A business line can receive up to a /56. These prefixes are dynamic and will change, just like a DHCPv4 address.

If you have a residential line and just one network internally, the default /64 will do fine.

If you have more than one network, you can give your ISP a “hint” that you’d like a /60 (16 networks) or /56 (256 networks, business line).

To clarify the underlying mechanics, DHCPv6 assigns a /128 address to your outside interface, and “delegates” a prefix that you can then use to assign /64s to your internal interface(s) as desired, or, indeed, delegate further inward to another router.

Here’s an example that’s requesting a /60 prefix. This example only shows the ipv6 portion of the configuration.

config system interface
  edit "wan1"
    config ipv6
      set ip6-mode dhcp
      set ip6-allowaccess ping
      set dhcp6-prefix-delegation enable
      set dhcp6-prefix-hint ::/60

If you don’t have more than one internal interface, you can leave the hint off. Comcast on a residential line will assign a /64 in that case, for example.

Assigning prefixes to internal interfaces and addresses to clients

I owe others for explaining how to do this, notably Myles and /u/iwanttoride . Without their explanations, I’d still be stuck thinking that FortiOS doesn’t support dynamic allocations. The examples given in the FortiOS handbook are brief and lack all explanation.

Before getting started, two decisions need to be made:

Which DNS servers will be used? Those of the ISP or others? If others, such as Cisco Umbrella / OpenDNS or CloudFlare’s privacy DNS, enter those servers, both IPv4 and IPv6, under Network -> DNS, and choose to “Specify” your own servers. I’m showing using specified DNS servers, and will mention the commands required to use the ISP’s DNS servers instead.

Will you assign addresses using DHCPv6, or use DHCPv6 for DNS assignment only? I am showing DNS only, DHCPv6-lite. I’ll mention the commands required if you want to use DHCPv6 address assignment, though I’m not sure what would be gained. DHCP monitor seems to show only IPv4.

Assigning addresses to clients is reasonably straightforward, though there are implementation differences. Some OSs will receive DNS via DHCPv6, others only through RDNSS. Some can receive addresses via DHCPv6, others, notably Android, can’t.

This means you’ll be presenting DNS via both DHCPv6 and RDNSS. Make sure the two match and deliver the same server(s). Likewise, if you use DHCPv6 for address assignment, make sure it matches the SLAAC assignment on the interface.

Here are the commands to use the first /64 of your delegated prefix on an internal interface. As before, I’m only showing the ipv6 portion of the configuration.

config system interface
  edit "lan"
    config ipv6
      set ip6-mode delegated
      set ip6-allowaccess ping https ssh
      set ip6-send-adv enable
      set ip6-other-flag enable
      set ip6-upstream-interface "wan1"
      set ip6-subnet ::1/64
      config ip6-delegated-prefix-list
        edit 1
          set upstream-interface "wan1"
          set autonomous-flag enable
          set onlink-flag enable
          set subnet ::/64
          set rdnss-service default

If you wanted to use DHCPv6 for address assignment, add “set ip6-managed-flag enable” to the “config ipv6” section. Because of OS implementation quirks, you should keep both the managed-flag and the other-flag in that case.

This may require a reboot. I am not sure of that, and it might depend on FortiOS version. I think FortiOS 6.0.1, where I tested that, just needs a couple minutes time to assign the interface its address, but I’m not 100% certain of that.

Keep in mind you can “get” the actual values on your interface once you are inside its configuration via “edit lan” or whatever your interface name is.

Let’s discuss these in some more detail.

We’re, obviously, in delegated mode, and we got a delegated prefix on our “ip6-upstream-interface”, in my case “wan1”.

ip6-allowaccess is for management access. If you have a FortiManager on this interface, or FortiAPs in tunnel mode, add the relevant services like you would in IPv4.

We’re sending RAs so that clients can learn an address and a next-hop, that’s ip6-send-adv. We’re telling clients they can get a DNS server (and if you feel like it, a dns-search-list) via DHCPv6, that’s “ip6-other-flag”.

The ip6-subnet is the part I didn’t understand for the past 2 years or so. This is a mask that is applied to the delegated prefix using a logical Boolean “AND” operation. In practice, you can think of this as just “adding” this value to the delegated prefix. “::1/64” means that we’ll take the delegated prefix as-is, and add “1” at the very right, then change the length to 64. This becomes the IPv6 address of this interface. Let’s say my delegated prefix is 2001:db8:3c4d:f40::/60, then the IPv6 address of my “lan” interface would become 2001:db8:3c4d:f40::1/64.

Now we need to hand out addresses to clients on this network. That’s what the “ip6-delegated-prefix-list” is for. I only need one subnet per interface, so “edit 1” it is. The “upstreams-interface” needs to be set, once more. “autonomous-flag” and “onlink-flag” tell the client that it can get an address via SLAAC here and that this interface has an address in that range, that is, can be routed to, respectively.

The “subnet” functions similarly to the “ip6-subnet” for the interface address. Here, we’d want to add to the network portion, not the host portion of the IPv6 address. Since I’m using the very first subnet in my allocation, there’s nothing to add. “::/64” combined with my 2001:db8:3c4d:f40::/60 prefix gives me 2001:db8:3c4d:f40::/64 to be used for client addresses on this interface.

Lastly, “rdnss-service default” will hand out my system DNS server(s) to clients on this link, if the client speaks RDNSS. That’s the belt and suspenders approach to DNS, discussed above. If you wanted to use the ISP’s servers instead, you’d configure “rdnss-service delegated”.

And here’s the DHCPv6 portion to hand out DNS.

config system dhcp6 server
  edit 1
    set dns-service default
    set interface "lan"

If you wanted to use the ISP’s DNS servers, that would become “set dns-service delegated”, and you’d add “set upstream-interface “wan1″”.

If you had DHCPv6 also hand out addresses, it’d look like this.

config system dhcp6 server
  edit 1
    set dns-service default
    set subnet ::/64
    set interface "lan"
    set upstream-interface "wan1"
    set ip-mode delegated

Now, let’s do this all over again for guest WiFi, assigning the second /64 subnet out of my /60 allocation.

config system interface
  edit "wifi"
    config ipv6
      set ip6-mode delegated
      set ip6-allowaccess ping
      set ip6-send-adv enable
      set ip6-other-flag enable
      set ip6-upstream-interface "wan1"
      set ip6-subnet ::1:0:0:0:1/64
      config ip6-delegated-prefix-list
        edit 1
          set upstream-interface "wan1"
          set autonomous-flag enable
          set onlink-flag enable
          set subnet 0:0:0:1::/64
          set rdnss-service default

Guests don’t need management access to my Fortigate, so “ip6-allowaccess ping” suffices here.

The ip6-subnet value of ::1:0:0:0:1/64 combines with my 2001:db8:3c4d:f40::/60 prefix to give this interface an IPv6 address of 2001:db8:3c4d:f41::1/64. Keep in mind that the left-most four fields of the address are the network, and the right-most four fields are the host.

For assigning addresses to clients, I’m matching this and set subnet to 0:0:0:1::/64. This is for the subnet, which means I care about the network portion only, hence the :: for the host fields. Combine that again with my delegated 2001:db8:3c4d:f40::/60 prefix and clients will receive addresses from 2001:db8:3c4d:f41::/64. That’s on-link to the interface address and they’ll be able to route.

Remember to create another DHCPv6 entry for DNS, and if you use DHCPv6 to give out addresses, make sure that entry’s subnet value matches the value here.

config system dhcp6 server
  edit 2
    set dns-service default
    set interface "wifi"

And with DHCPv6 handing out addresses:

config system dhcp6 server
  edit 1
    set dns-service default
    set subnet 0:0:0:1::/64
    set interface "wifi"
    set upstream-interface "wan1"
    set ip-mode delegated

Rinse repeat for any other interfaces you may have. A /60 delegated prefix gives you 16 available subnets, from 0 through f.

IPv6 firewall policy

Time for something simpler. All you need here is a policy allowing ICMPv6 in and out, and a policy for traffic out to the Internet or other subnets.

I have multiple interfaces so I enabled “Multiple Interface Policies” in “System -> Feature Visibility”, to make my life easier.

I’d like to follow RFC 4890 instead of allowing “all ICMPv6”. ICMPv6 is needed for session health, and I still want to be security-conscious.

config firewall service custom
  edit "ICMP6-DestUnreach"
    set category "General"
    set protocol ICMP6
    set icmptype 1
    unset icmpcode
  edit "ICMP6-PacketTooBig"
    set category "General"
    set protocol ICMP6
    set icmptype 2
    unset icmpcode
  edit "ICMP6-TimeExceeded0"
    set category "General"
    set protocol ICMP6
    set icmptype 3
    set icmpcode 0
  edit "ICMP6-TimeExceeded1"
    set category "General"
    set protocol ICMP6
    set icmptype 3
    set icmpcode 1
  edit "ICMP6-ParmProb0"
    set category "General"
    set protocol ICMP6
    set icmptype 4
    set icmpcode 0
  next  edit "ICMP6-ParmProb1"
    set category "General"
    set protocol ICMP6
    set icmptype 4
    set icmpcode 1
  edit "ICMP6-ParmProb2"
    set category "General"
    set protocol ICMP6
    set icmptype 4
    set icmpcode 2
  edit "ICMP6-EchoRequest
    set category "General"
    set protocol ICMP6
    set icmptype 128
    unset icmpcode
  edit "ICMP6-EchoResponse"
    set category "General"
    set protocol ICMP6
    set icmptype 129
    unset icmpcode
config firewall service group
  edit "ICMP6-allow"
    set member "ICMP6-DestUnreach" "ICMP6-EchoRequest" "ICMP6-EchoResponse" "ICMP6-PacketTooBig" "ICMP6-ParmProb0" "ICMP6-ParmProb1" "ICMP6-ParmProb2" "ICMP6-TimeExceeded0" "ICMP6-TimeExceeded1"
    set comment "ICMP6 services to be allowed as per RFC4890"

With that group created, you can now create your first three IPv6 policies.

Allow desired ICMPv6 and drop all other ICMPv6:

From “any” interface to “any” interface, source and destination “all”, schedule “always”, Service “ICMP6-allow”, action “Accept”, and no NAT. I turn logging off as well.

From “any” interface to “any” interface, source and destination “all”, schedule “always”, Service “ALL_ICMP6”, action “Deny”. I turn off logging.

Allow your interfaces out to the Internet:

From internal interface, say “lan”, to external interface, say “wan1”, source and destination “all”, NAT disabled, action “Accept”, and Schedule, Service, Security Profiles and Log as desired. Add additional “from” interfaces if they share the same policy.

Testing, References

IPv6 connectivity testing is available at ipv6-test and test-ipv6.

Myles documented prefix delegation in a way I could understand.

/u/iwanttoride explained an example configuration of PD on reddit.

Antonios Atlasis and Enno Rey lab-tested client implementation differences in receiving an address and DNS server.

James Sanders explained Android’s requirement for RDNSS.

E. Davies and J. Mohacsi wrote RFC 4890, a recommendation on how to filter ICMPv6 messages on a firewall while still allowing IPv6 to function.

JunOS SPACE installation notes

Installing JunOS SPACE can be a slog through documentation. These are my notes to help with the needed steps.

Edit: vSphere 6.5 can have issues installing the OVA file provided by Juniper. Until Juniper provides OVF files, you can install ovftools and convert the OVA using those.

ovftool -st=OVA -tt=OVF space-17.1R1.7.ova space-17.1R1.7.ovf

Source (.ova) and destination (.ovf) paths to be adjusted by you as needed.

-tt – Target Type (Explicitly express that the target is OVF, OVA, VMX, VMX, vSphere, vCloud, ISO, FLP, or vApprun)

-st – Source Type (Explicitly expresses that the source is OVF, OVA, VMX

Assuming SPACE is going to be installed on VMWare, as version 16.1 or better, this is the recommended sizing:

32GB RAM (OVA installs as 8GB, increase it)
4 vCPU
~ 500GB storage space total

The OVA installs as about 250G storage space. 100G of that is /var. You’ll want to add to that. Depending on the size of the environment being managed and whether this node will also handle integrated logs, anywhere from 250GB to 1TB of additional space is appropriate. It’s possible to add 250GB to start with and then add additional space if required.

SPACE can be deployed as a cluster, all in the same subnet, as well as in a DR scenario across L3 boundaries. Most of my customers run a single instance, as their environment is not large enough to warrant cluster deployment and they can rely on VMWare for DR purposes.

SPACE requires two IP addresses – one for the physical node and one for the VIP, used for HTTPS GUI access. Any additional nodes would need one additional address in the same subnet.

SPACE can use a second interface to communicate with devices if desired. This can be handy if the device management interface and the GUI access need to be in separate subnets.

SPACE does not offer a supported way of firewalling itself. You’ll want to firewall it in your environment, at a minimum restricting access to internal subnets, better yet restricting access to trusted subnets. This is a list of the services used, subject to revision should I miss a few. Juniper have a KB article on this which might be more accurate.

– HTTPS inbound for GUI access. Optional Ping inbound.
– If you are using eth0 for device management (no dedicated device management interface), and you don’t have a dedicated monitoring node: SNMP Trap inbound

Physical IP:
– ssh inbound for admin console access. Optional Ping inbound.
– DNS, NTP and SMTP outbound to your DNS/NTP/SMTP servers. RADIUS / TACACS+ outbound to your AAA server(s), if configured. Optional Ping outbound.
– HTTPS and SSH outbound to “*”, or if deploying as a customer instance connecting back to a Juniper partner, * and the partner SPACE proxy. When in doubt, this can be HTTPS outbound “to the Internet.” Optional Ping outbound.
– SNMP inbound if you are using an SNMP monitoring solution to monitor JunOS SPACE itself
– If you have a SPACE cluster with several nodes, they’ll communicate on that subnet using multicast. If multicast does not function in your environment, you can switch to unicast. I’m not sure what the implications of doing so are and prefer to run the default multicast configuration.

Physical IP or Device Management IP, if it was configured:
– ssh, ping and snmp outbound to device subnets
– ssh on port TCP-7804 inbound from device subnets
– snmp-trap inbound from device subnets, if the option to configure SNMP traps upon device discovery is set

If you have a dedicated device monitoring node, snmp-trap will be sent to it. If some of your devices will reach SPACE through NAT, you’ll want to read Juniper’s guidance on it.

These are the parameters you should have before you install SPACE:

  • DNS server address
  • NTP server address
  • Time Zone
  • VIP IP
  • Physical IP
  • Gateway IP
  • If SPACE will be behind a NAT device for device access, you’ll need to specify that during setup and have the NAT addresses handy
  • Node Name – for display in GUI, not the FQDN
  • admin password
  • maintenance password
  • super password
  • Desired FQDN – for DNS entry you will create
  • Cert for that FQDN – to avoid security warnings when connecting to GUI. This can absolutely be an internal trusted CA
  • Desired user authentication method – your options are Local, RADIUS and TACACS+
  • user name and password – for download of schemas and setup of ASAP, the pro-active service app
  • Any apps you will install such as ASAP (aka Service Now), Security Director, Network Director
  • If using Security Director or Network Director, license authorization keys for those and JS-PLATFORM. You’ll “cut” the license once SPACE is installed. If only using ASAP (ne Service Now), you will receive a permanent license once you connect SPACE to, free of charge as long as you have at least one device under active maintenance.
  • SMTP server (and any authentication you need) for SPACE to send email
  • Username and password (or ssh key) you will use to manage devices. A “service account” such as “spcadmin” is often a good idea. This account needs “admin” rights in JunOS.
  • SNMPv3 or SNMPv2 read-only for use by SPACE. Optional (but recommended), allows Network Director and OpenNMS to monitor devices.

Once the OVA is installed, has been increased to 32GB RAM and has had an additional disk allocated to it – do NOT increase the size of the disk the OVA creates for you, that won’t work – you are ready to power it up and go through initial configuration. It’s pretty straightforward and asks for many of the parameters you gathered above.

The default username for console access is admin / abc123

The default username for GUI access is super / juniper123

When SPACE is up and running, you’ll have an option to expand virtual drive space. Choose it and assign all the additional disk space your VM guest received to /var. Be careful with this, if you accidentally assign it to another partititon (such as /, /tmp or /var/log), you’ll need to wipe the VM and start over.

Save your admin, maintenance and super passwords in an encrypted, centralized, backed up password safe. You can recover if you lose the admin password, but it takes access to the VMWare host and some effort.
“admin” is used for ssh access, “maintenance” is used for upgrades of the JunOS SPACE platform itself, and “super” is used for GUI access.

Additional housekeeping while on console and waiting for jboss and thus the GUI to start:

  • Change admin password expiration. Default is 70 days; you’ll likely want a longer timeout or “never”.
  • Change ssh session timeout. Default is 5 minutes. You can edit /etc/ssh/sshd_config and set ClientAliveInterval to 600 and ClientAliveCountMax to 3 and you’ll have 30 minutes.
  • Install VMWare Tools. Your VMWare admins will thank you.

If you want to avoid HTTPS security warnings when connecting to JunOS SPACE, create a DNS entry for its VIP address with the FQDN you chose, then create a cert (again, trusted internal CA is fine) for that. You’ll load that in the GUI under Administration -> CA/CRL certificates. Load the cert and any needed intermediate CAs.

Once in the GUI, you’ll want to change some of the default settings. Go to Administration -> Applications, right-click “Network Management Platform” and choose “Modify Application Settings”.
– “Allow Device Communication” is critical.
– “Add SNMP configuration to device for fault monitoring” can be useful if you want to use OpenNMS, but isn’t critical.
– “Configure commit synchronize” creates issues with single EX devices, uncheck that.
– “Manually resolve fingerprint conflicts” is probably more hassle than its worth for all but the most security-conscious customers.
– “Auto Resync”, “Approval workflow” and “commit confirmed” are useful
-Under “User”, set the timeout. 30 or 60 minutes seems reasonable for most environments.
– Under “Password”, set the password expiry in months. I’ve seen customers set this to “120” because they believe in the revised NIST guidelines and prefer good passwords over frequent changes.
– Under “Security”, the “Disable weak algorithms” checkbox will help the device pass an audit.

And hit “Modify”, wait for JunOS SPACE to restart its web server, and log back in.

If you are not going to use OpenNMS, you may disable it under Administration -> Applications -> Network Management Platform -> Manage Services

Under Administration -> DMI Schemas, set SPACE up to be able to pull DMI Schemas.
Click on the “Update Schema” icon, click the “SVN Repository” radio button and the “Configure” button. The URL is, the username and password are a login that belongs to the organization running this SPACE instance. “Auto Install Schema” is a good idea as it avoids additional work. “Test Connection”, then “Save”.

Under Administration -> SMTP Servers, set up your mail server.

Under Administration -> Authentication Servers, set up your RADIUS/TACACS+ auth. I recommend “Remote-Local Authentication” so that you can still get into the unit using “super” if the remote authentication fails.

Under Administration -> Database Backup and Restore, you can set a backup to an scp server. It’s likely you’ll be relying on VMWare snapshots, but if you don’t have that in place, this is highly recommended.

Under Administration -> Purging Policy, set a policy to purge disk space periodically. Not really needed unless you take regular local DB backups or have very large device configuration files, in which case it becomes critical.

Under Administration -> CA/CRL certificates, install your HTTPS certificate.

Under Administration -> Fabric, enable the Cassandra service using Actions -> Enable Cassandra. This improves MySQL performance by offloading device image files to the Cassandra service.

Install any applications you’d like to use. ASAP (ne Service Now) is quite useful, and Security Director is the obvious choice for SRX policy management.

When deploying Security Director, I recommend also deploying a second node as Log Collector.  Unless you already have the SIEM IBM QRadar or Juniper JSA collecting logs, in which case you can just point Security Director towards those.
Log Collector will require another 16GB, 2 IP addresses (one in the same subnet as the main SPACE node for cluster comms and one for syslog, can be in the same subnet but need not be), and either 500GB of disk space and an NFS share, or 1TB (or more) of disk space to hold logs locally.

If you do use ASAP (Service Now), here are a few settings that’ll help you out:

You’ll add an “Organization”. If you are going through a partner proxy with PAR service instead of direct to Juniper, work with the partner on that setup. They’ll point your instance to their proxy, load a certificate file for their proxy, and they may set an auto-submit policy for incidents.

Administration -> Global Settings -> Core File Upload Configuration, set this to “Secure FTP upload through Service Now”. Otherwise devices will try to FTP directly to and that will likely fail.

Here are some good videos by Juniper on using ASAP:

Video #1:
Video #2:
Video #3:
Video #4:


Lastly, when adding devices to SPACE, consider assigning them a public tag such as “All Devices” and configuring Configuration Files backup to act on that tag on a schedule, say once a day.

You can create schedules to find new devices automatically, and you can of course use the base JunOS SPACE application to upgrade firmware and make bulk configuration changes.

JunOS SPACE upgrade to 16.1r2

These are my notes for upgrading JunOS SPACE from 15.2r2 to 16.1r1 or 16.1r2. They are meant to be consumed together with Juniper’s upgrade instructions. Since you are installing a fresh copy of JunOS SPACE as part of this upgrade, maybe now is also a good time to revisit some default settings.

  • 16.1 is the first release where the default partition sizes in the OVA are “sane”. The only partition you’ll need to add to is /var. It is 100GB large by default. An additional 250GB is fine for most installations; large installations with massive DBs might want as much as an additional 1TB.
  • You may not have enough space on the disk to take a backup using the 15.2r2 backup patch as long as OpenNMS remains enabled. In that case, disable it; then after reinstall and import, take additional steps to re-enable it. Disabling OpenNMS is done from Administration -> Applications -> Network Management Platform -> right-click and Manage Services
  • When taking the backup, I then opted not to backup PGSQL (that’s OpenNMS) and FMPM (since I happen not to have any FMPM nodes). This reduced the size to something manageable.
  • You may need the ServiceNow image file when taking the backup. If so, copy it to SPACE using command-line scp, then move it to /var/cache/jboss/jmp/Service-Now.VERSION (the backup process will tell you the exact location), and hit Enter to let the backup continue. For Service Now 16.1r1, the location is /var/cache/jboss/jmp/Service-Now.16.1R1.15
  • You will require an external scp server to copy the backup file to, or you can use a USB stick with FAT32 (no more than 32GB) if upgrading a JA2500 appliance.
  • Front USB is detected as /dev/sdb, use dmesg to make sure. Then mount:
    mkdir /tmp/pendrive
    mount -t vfat /dev/sdb1 /tmp/pendrive
    You can check with fdisk -l
  • To restore from USB, go through initial configuration.  When you come to restore choice (Remote, USB, Local), ssh to device and mount -t vfat /dev/sdb1 /tmp/pendrive, then use serial console to choose USB
  • If you don’t have an scp server, you can choose during backup; then copy the file over to the new server during restore and choose “Local”
  • You will need these things to configure your new SPACE instance:
    DNS server
    NTP server
    VIP IP
    Phys1 IP
    Phys2-N IP
    GW IP
    License File
    List of Apps
    admin password
    maintenance password
  • After restoration, adding space to /var etc, check the settings for Network Management Platform. “Allow device communication” may be off. Turn it on so devices will move to “Up” status.
  • For a secondary node, it doesn’t ask for NTP on initial setup. Set this and TZ manually. Once secondary node is up, you’ll need to add it from GUI as well.
  • Don’t forget chage admin and the ClientAliveInterval / ClientAliveCountMax in /etc/ssh/sshd_config
  • If you disabled OpenNMS before the backup, it won’t start after import. This is how you get it back in a default state.
    Disable OpenNMS from GUI
    service postgresql-9.4 status
    If it’s down: service postgresql-9.4 start
    Now to create the DB:
    service jmp-watchdog stop
    service jmp-opennms stop
    For the following, passwords are postgres and opennms respectively
    psql -U postgres -c ‘ALTER ROLE opennms SUPERUSER’
    psql -U opennms postgres -c ‘drop database opennms;’
    psql -U opennms postgres -c “create database opennms encoding ‘unicode'”
    psql -U postgres -c ‘ALTER ROLE opennms NOSUPERUSER’
    /opt/opennms/bin/install -dis
    service jmp-watchdog start
    Then enable OpenNMS from GUI


JunOS SPACE fails when upgrading applications

When upgrading JunOS SPACE, the applications installed on it need to be upgraded as well. I’ve seen jboss crash and restart when several applications are upgraded in a row. This happened after moving to SPACE 16.1r2 and then again when upgrading to 17.1r1.

JTAC advised that the issue is that Java runs out of “PermGen” memory. Assuming that SPACE is installed on a JA2500 or a VM with 32GB of memory, these changes should resolve the issue. They may have to be reapplied after each upgrade of the core SPACE software.

  • stop jboss and watchdog
    service jmp-watchdog stop
    service jmp-watchdog status (make sure it has stopped)
    service jboss stop
    service jboss status (make sure it has stopped)
  • edit /var/jboss/domain/configuration/host.xml.slave and change
    <option value=”-XX:MaxPermSize=512m”/>
    <option value=”-XX:MaxPermSize=1025m”/>
  • start watchdog
    service jmp-watchdog start
    service jmp-watchdog status

    The watchdog process will restart jboss so there is no need to restart jboss manually.