Recovering from failures after the event

Shortly after the rollover time, we expect affected devices to start misbehaving. If misbehaving devices are power cycled they will likely return to proper operation and not need to be power cycled again until the next rollover event (in four years).

The rest of this page are specifics about how to reboot a SyncBox connected in various ways. If you are unable to reboot the device in a timely manner for some reason, you should visit the mitigation page for some suggestions as to how you might be able to temporarily work around this issue.

Info

As this issue is caused by a very specific rollover event, it is a good idea to verify that the rollover event has already occurred.

The rollover occurs at midnight Moscow time as 2024 starts. So, if it is not yet 2024 in Moscow, then the rollover has not occurred, and any failures are not likely the result of this issue.

The easiest and most reliable way to verify that the rollover has occurred is to do an internet search for "what time is it in moscow". If the date is in 2024, the rollover has ocurred and you should proceed with the steps on this page. On the other hand, if rollover has NOT occurred yet, your issues are likely due to some other issue.

Devices receiving "uGPS power" from the Aux port or timing port on the radio

Most modern cambium radios in the 450 line (and possibly some earlier ones) can provide power to an attached GNSS receiver. PacketFlux has released the "SyncBox Aux Port Version" product which allows up to four radios to be connected to a single device.

Because the SyncBox can receive power from any of the devices that it is attached to, it is necessary to cause all of the radios attached to each Aux Port receiver to stop providing power at the same time. If any of the radios keep providing power, the SyncBox will not restart.

If there is only one radio attached, in most cases it is only necessary to reboot the single radio providing the power.

Where multiple radios are attached, it becomes a bit more complex. In some cases, customers have been able to simultaneously reboot all of the radios at almost exactly the same time to remove power. This works if the reboot command can be issued to all the radios in a very short period and if the radios remove power for a long enough period that all of the radios have turned off uGPS power at the same time.

A safer method is to switch the uGPS power off in all of the radios (rebooting each as you go). After all of the radios have rebooted with uGPS power off at the same time, you can then re-enable uGPS power in at least one radio to restore power and get sync working again. A slight variation of this is to turn off uGPS power in all but one of the radios and then rebooting the final radio to cause the SyncBox to reset. The uGPS power setting is on the Configuration->General tab and is labeled "Aux Port Power to UGPS"

Devices connected to a RackInjector or to a SyncInjector or PowerInjector connected to a Base 3

On the RackInjector, rebooting an attached SyncBox is easy. Navigate to the GNSS and 1PPS tab in the web interface on the RackInjector. Click on the GNSS Port Power button to disable the power on the port, then once a few seconds have elapsed, click on the button again to re-enable power on the port. Note that if you have multiple RackInjectors at a site, you'll need to cycle power on the RackInjector the SyncBox is directly connected to.

On SyncInjector or PowerInjector+Sync products attached to a Base 3, the procedure is similar. The only difference is that you'll need to select the correct PowerInjector first. Navigate to the GNSS and 1PPS tab, select the PowerInjector or SyncInjector the SyncBox is attached to, and then click on the GNSS Port Power button to disable power to the port. After it has been off for a few seconds, click on the button again and re-enable power to the port. If you're not sure which unit has the SyncBox attached directly to it, then you'll need to repeat this on each PowerInjector or SyncInjector.

Note that while the power is off you should not see the GNSS UTC Date and Time incrementing. Once you reestablish power, the Receiver Restarts counter should increment. Seeing this occur is a good indication that the power did, in fact, get cycled.

Devices connected to a SyncInjector or PowerInjector connected to a Base II

Once you log into a Base II, you'll need to navigate to the Expansion Units tab. On this tab make note of the index for the SyncInjector or PowerInjector+Sync that has the SyncBox directly attached to it. If you do not know which device has the SyncBox attached, you'll need to repeat this for each device it might be attached to.

Click on the Binary I/O tab. If you look at this table you should find a section where the slot number matches the index you found in the previous step. Look in this section and hopefully you'll find a row with a description which says "SyncPipe Power". Click on the Cycle column (it should be the next to the last value in the row with a pencil beside it). In the popup window, enter the Value '300' to turn off the power for 30 seconds then turn it back on. If you need the power off for a longer or shorter time, you can modify this value accordingly.

If you'd like to verify the power is actually off, you can look on the Analog tab to verify that the "Used Pulses" isn't counting up during the period where the SyncPipe power is off.

In the event that the "SyncPipe Power" label has been overwitten in the past, you may be able to determine which row applies based on elimination. On the unit I'm using to verify these instructions, it is port 12. It may be slightly higher or lower depending on the version of the injector you are using.

Devices connected to a SyncInjector or PowerInjector connected to an original base unit

The original base unit did not have a web interface. Instead it uses the "SiteMonitor Manager" windows-based tool available from our support website.

Once you obtain and install that tool, the method of operation is very similar to the Base II web site. As a result you can follow the instructions found in the Base II section above, with the following main differences:

• You'll need to enter the ip address and SNMP read-write community string for the device at the top of the application.
• The SiteMonitor manager does not refresh itself. As a result, you need to click Refresh whenever you'd like to obtain the values from the remote unit.
• The Binary tab in the Base II is labeled "Boolean I/O" in the SiteMonitor Manager.
• There are subtle differences in the interface. For instance, the column values do not have a pencil to indicate that they can be changed.

SyncBox12's powered via the parasitic port option

On the SyncBox 12 there is an option to pass the PoE connection from one of the radios through the SyncBox's Parasitic PoE passthrough port. In this case, you'll need to cycle the power for both the SyncBox and the Radio which the SyncBox is borrowing power from.

Because the specifics of how to do this will vary from network to network, it's impossible to cover all of them here. In general, this will usually involve cycling power on the PoE injector that the radio is being powered from - whether that is a PacketFlux PoE injector or some other device.

SyncBoxes powered through other means

The above covers most of the methods which our customers use to power SyncBoxes, but we are aware of some other methods that are in use. The most common methods use a separate injector or spare passive PoE port or a custom cable or similar to provide power to the SyncBox power port. Because these methods will vary with the exact method used, the exact method to cycle power will also vary.

We know that there will likely be some of our customers who have inherited a network which was built by someone else. In that case, our main guidance would be to take the time to understand how the SyncBox is powered at each site and to come up with a strategy on how to power cycle it.