Merging a Test Dataset into a Production Dataset (ubisense_ls_dataset_merge)

Introduction

It is a common requirement, especially in car assembly plants, to extend existing production sensor installations. Normally this will mean that there will be a partitioning of the assembly plant space into sub-spaces where Ubisense sensors are in live production and sub-spaces where Ubisense sensors are still being commissioned. For DIMENSION4 this can be done using two separate datasets, one for the existing production system and one for the new sensors in test.

This document describes the method for merging two datasets.

  • The Overview gives a high-level overview of the extension and merging process.

  • Method describes the method for extending the system with new sensors on a test system and then merging the extensions into a live production system.

The merge process uses the ubisense_ls_dataset_merge command-line tool. This tool allows the merging of all sensors or sensors selected by sensor group, MAC number or searches. An invert option allows all but some specified to be merged.

See Examples of Selecting Sensors for Merging for further information.

Note: By default, all 2.4 GHz receivers will be merged.

Audience

Field engineers involved in the expansion of a DIMENSION4 installation.

Installation

To get the tool, run Application Manager, open the DOWNLOADABLES task, and expand Location system /Administration tools. Select ubisense_ls_dataset_merge and click Download selected items. Optionally, change the download destination directory, and then click Start download.

Overview

Let us suppose there are two servers: PROD is the current production server, TEST is the server for commissioning new sensors. We will call the datasets for these servers PROD and TEST.

The process for extending a system involves two main stages:

 

  1. Commissioning the RTLS on TEST while the system is live on PROD and then merging the changes back to PROD, preserving any changes that were made to PROD during the commissioning process. This means the user can make changes to PROD during this stage and these will be preserved by the merge.

  2. Commissioning the application, which can be done in one of two ways:

    • Commission on TEST while the system is live on PROD and then copy the changed dataset back to PROD.

      Changes made to PROD during this stage will be overwritten when the dataset is copied back from TEST.

    • Commission directly on PROD, ensuring that essential changes made to PROD are not lost.

      This has the downside that commissioning work is done directly on a live system.

Application commissioning (stage 2, above) is not discussed further in this document. RTLS commissioning is discussed in detail.

The RTLS commissioning process in more detail looks like this:

 

Stage number

Stage

PROD server

TEST server

Production status

1

Initial state

 

In production

 

Live

2

Preparation

Reconfigured for side-by-side operation

 

 

Stopped

3

Copy PROD to TEST

 

Copy of PROD installed ready for RTLS sensor commissioning

Stopped

4

Commission RTLS on TEST

In production, some changes can be made on live system if needed

Changes to RTLS configuration for system extension

Live

5

Merge TEST with PROD

TEST changes merged with any changes made to PROD during stage 4

 

Stopped

6

Repeat stages 3 to 5 if required

 

 

 

Live while commissioning

Method

Preparation stage

To start with, it is necessary to set up PROD and TEST with suitable datasets for side-by-side operation to occur.

This means that it needs to be guaranteed that:

  • Commissioned sensors will use the configuration server on PROD when they boot

  • Non-commissioned sensors will use the configuration server on TEST when they boot

  • Services running on PROD will use the configuration server on PROD

  • Services running on TEST will use the configuration server on TEST

For ease of understanding we will assume that the PROD system is set up in ‘no_multicast’ mode. Other modes are possible and can be set up to support side-by-side operation, but it’s most likely that the initial target site will be in ‘no_multicast’ mode. In this case, we are assuming that there is one server machine for PROD and one for TEST, and the IP address of PROD is IPP and the IP address of test is IPT.

STEP 1: The complete system needs to be set up like this:

  1. The ubisenseconfig DNS entry should point to IPT — this means that any new unknown sensors plugged in will be directed to the test system and not the production system.

  2. All commissioned sensors must have a static IP address and a configuration server search path that points to the PROD server machine – this means that commissioned sensors will connect to the production system when they boot.

  3. There should be a proxy configuration entry (IPP/32 à IPP) to ensure that services on PROD locate the configuration server on PROD. This is only strictly required on TEST but for convenience and safety when copying datasets it should be on both servers.

STEP 2: The TEST dataset must be constructed starting from a copy of the PROD dataset.

This means all the objects which are common to both datasets must be exactly the same with the same unique Ubisense IDs.

STEP 3: The TEST dataset should be moved to the TEST server machine.

The TEST and PROD datasets should run on separate server machines.

STEP 4: The IP addresses of all the commissioned sensors in the TEST dataset must be zeroed so they do not reference the PROD server.

  • Zero the sensor IP addresses on TEST using the ubisense_ls_config_admin.exe zero-sensor-ips command.

Post-conditions

At the end of the preparation stage the following should hold true:

  1. If a new sensor is plugged in then it should connect to the TEST configuration server

  2. When commissioned, production sensors are rebooted then should connect to the PROD configuration server

  3. Production dataset services should connect to the PROD configuration server and test services should connect to the TEST configuration server

  4. The TEST dataset should not contain any IP addresses for PROD

STEP 5: Test that these 4 conditions hold true.

RTLS commissioning stage

Commissioning new sensors

Commissioning may be broken into more than one phase. For example, suppose that Line 1 is already commissioned and in production. During the extension phase, sensors for Lines 2 and 3 will be added. These may be broken down into separate phases that run in series or parallel, or overlap. The two lines can be merged into the production system separately and at different times.

STEP 6: On TEST it is possible to go through the complete sensor commissioning, logging, analysis and optimization steps, including:

  • Creating new cells

  • Placing sensors, 2.4GHz receivers and TDUs

  • Configuring sensor networking and software

  • Solver runs

But it is important to remember that any change that would contradict the PROD will be lost, for example:

  • Changes to tag filters used by PROD (though it is fine to add a filter to a new tag or tag range)

  • Changes to properties of commissioned sensors

  • Changes to extents of cells in PROD

It is also important to note that:

  • If you accidentally delete from the commissioned part of the system, then do not try to create new data to replace it – this may well be regarded as new data. In practice this is not a hard rule to follow.

  • If new sensors are added to existing cells causing the cell extent to change, then the new extent will not be merged.

  • If the site cell extent needs to be changed then the new extent will not be merged.

Sensor swap out

You can carry out a sensor swap out on the production system during the extension phase, but you need to perform the swap in both PROD and TEST datasets, to avoid the sensor being associated with both MAC addresses in the resulting dataset.

Changes to sensor properties

Changes to the properties of PROD sensors in the TEST dataset are supported as long as they don't contradict existing rows in PROD dataset tables. For example, you can add a sensor that was already in PROD to a new group that is only in TEST, and it is possible for that to be successfully merged.

Preparation for dataset merge

Once a new set of sensors has been commissioned they need to be prepared for the merge phase. Using our earlier example, if Lines 2 and 3 are being commissioned and Line 2 is ready to be merged then all the sensors for Line 2 must be prepared for the merge.

STEP 7: All the newly commissioned sensors and TDUs to be merged must be given a suitable static IP address and a configuration server search path that leads to the PROD server machine. The sensors to be merged should then be rebooted.

  • They will locate the production server and set their search path to ‘valid’ in the local ROM, but then fail to boot from the production server (because they are not configured in it) and just continually reboot until after the datasets are merged and the new one brought up on the production server.

Note: The sensors must be given sufficient time to load the custom network configuration and also to ensure this is not invalid. There is a 12 minute timeout period on invalid network configurations. Allow a minimum of 30 minutes. Check the sensors do not reconnect to TEST server.

Dataset merge stage

At the dataset merge stage, we use a command-line tool that works over the two data directories for the two servers and merges the TEST data into PROD. TEST will be unchanged and PROD will be modified.

STEP 8: Backup all services in both the PROD and TEST datasets before commencing the merge.

STEP 9: In PROD the following services must be undeployed so that changes to the udata files are effective:

  • Platform – Cell extents

  • Platform – Fallback location cell data

  • Platform – Representation configuration

  • Location system – Configuration

STEP 10: Stop both the PROD and TEST datasets.

This means stopping the core server and service controller on each dataset.

STEP 11: Copy the PROD dataset to the TEST server.

STEP 12: Run the ubisense_ls_dataset_merge command-line tool to merge the two datasets.

When you run the tool, you must supply the PROD and TEST dataset paths as arguments and specify which sensors to include. You can run the command first in a “test” mode that prints a summary of objects and rows to be merged. For example

ubisense_ls_dataset_merge all-sensors C:\ubisense\prod C:\ubisense\test

summarizes the objects and rows that can be merged if all sensors in TEST are merged into PROD.

Options allow you to include only some sensors in the merge process, selected by sensor group, by specified MAC address, by search, or sensors with a custom network configuration. This selection can also be inverted to include all but the specified sensors. See Examples of Selecting Sensors for Merging for additional information.

You can also use the --check-sensors option to print the MAC addresses selected by the command to stdout without performing any other calculations.

Notes:

  • By default, all 2.4 GHz receivers are included in the merge, but can be excluded by using the --exclude-2.4GHz-receivers option.

  • If new sensors are merged (for example by using the by-group option), any new TDUs are not automatically included. Any TDUs added to the TEST dataset must be explicitly merged to PROD using the by-sensor option.

STEP 13: When you are ready to merge, run the command with the --commit option. For example:

ubisense_ls_dataset_merge all-sensors --commit C:\ubisense\prod C:\ubisense\test

Check progress messages and particularly look out for any warnings and errors which should always be investigated.

Note: You can run ubisense_ls_dataset_merge multiple times to merge the precise combination of sensors from TEST to PROD. To avoid errors, you must complete all merge operations before moving on to the next steps and restarting the production dataset.

STEP 14: Copy the PROD dataset back to the PROD server.

Rename the pre-merge version of the PROD dataset before copying back the post-merge version.

STEP 15: Start the PROD dataset.

Do not restart the PROD dataset until you have completed all merge operations.

STEP 16: Deploy the services which were undeployed in Step 9.

STEP 17: Check that everything has been merged as expected.

  • Ensure all services are running for all new cells.

  • Correct the sensor cell and site cell extents, if they were changed in TEST

  • Check that background objects have appeared correctly.

STEP 18: Test the merged system fully.

Examples of Selecting Sensors for Merging

Merging all sensors

This example merges all sensors from TEST to PROD including any 2.4 GHz receivers:

ubisense_ls_dataset_merge all-sensors --commit C:\ubisense\prod C:\ubisense\test

Merging all sensors except 2.4 GHz receivers

This example merges all sensors from TEST to PROD, excluding 2.4 GHz receivers:

ubisense_ls_dataset_merge all-sensors --exclude-2.4GHz-receivers --commit C:\ubisense\prod C:\ubisense\test

Merging sensors from a specific group

This example merges all sensors from a single group from TEST to PROD:

ubisense_ls_dataset_merge by-group --commit group1 C:\ubisense\prod C:\ubisense\test

Sensors in the group named “group1” are included in the merge. Sensors in other groups are not included.

Specify the groups to include in a semi-colon (;) delimited list.

You can obtain a list of available groups by using the following command:

ubisense_ls_dataset_merge list-groups C:\ubisense\prod C:\ubisense\test

Excluding a sensor group from a merger

This example merges all sensors except those in the specified group from TEST to PROD:

ubisense_ls_dataset_merge by-group --commit --invert group1 C:\ubisense\prod C:\ubisense\test

Sensors in the group named “group1” are excluded from the merge process. Sensors in other groups are included.

Merging sensors by listing MAC addresses

This example merges the two sensors given in the command from TEST to PROD:

ubisense_ls_dataset_merge by-sensor --commit 00:11:CE:01:58:3D;00:11:CE:01:58:3E C:\ubisense\prod C:\ubisense\test

Specify the sensors to include in a semi-colon (;) delimited list.

You can obtain a list of available sensors by using the following command:

ubisense_ls_dataset_merge list-sensors C:\ubisense\prod C:\ubisense\test

Merging sensors by search

This example merges sensors with a named search order (defined in Sensor network configuration in Location System Config) from TEST to PROD:

ubisense_ls_dataset_merge by-search --commit search1 C:\ubisense\prod C:\ubisense\test

Specify the searches to include in a semi-colon (;) delimited list.

You can obtain a list of available searches by using the following command:

ubisense_ls_dataset_merge list-searches C:\ubisense\prod C:\ubisense\test

Merging sensors with a custom network configuration

This example merges all sensors with a custom network configuration (defined in Sensor network configuration in Location System Config) from TEST to PROD:

ubisense_ls_dataset_merge cnc-sensors --commit C:\ubisense\prod C:\ubisense\test