Support migration from the Firefly LNS #97

KrishnaIyer · 2023-11-01T10:12:13Z

Summary

Support migration from the Firefly LNS. Replaces #49.

Changes

Improve on the ground work in Support migration from Firefly #49
Make the migration/exporting work.
Improve code reuse.

Testing

With a The Things Uno.

Curl the device using the API (to check the last frame counter)

curl "https://xxxxxxx/api/v1/devices/eui/0004A30B001BB199?auth=$FIREFLY_API_KEY"
{"device":{"updated_at":"2023-10-31T18:30:53.110455","tags":[],"skip_fcnt_check":false,"rx2_data_rate":0,"region":"EU868","override_location":false,"otaa":true,"organization_id":226,"organization":{"name":"alec","id":226,"description":null},"network_session_key":"1BA000AB1BAC28B83F2A091F7EA9ABED","name":"Migration Test OTAA","multicast_groups":[],"location":null,"inserted_at":"2023-10-31T18:11:08.544913","id":62268,"frame_counter":0,"eui":"0004A30B001BB199","device_class_id":null,"description":null,"deduplicate":false,"class_c":false,"application_session_key":"927A87F8B710479D594440227BBBDD64","application_key":"5105D918B6EEEF8A22AA92B39D9EC718","adr_limit":null,"address":"29559735"}}

This can also be checked via the UI

Export

./main firefly device 0004A30B001BB199 --invalidate-keys

{"ids":{"device_id":"eui-0004a30b001bb199","application_ids":{"application_id":"my-test-app"},"dev_eui":"0004A30B001BB199","join_eui":"1111111111111111"},"name":"Migration Test OTAA","lorawan_version":"MAC_V1_0_2","lorawan_phy_version":"PHY_V1_0_2_REV_B","frequency_plan_id":"EU_863_870","supports_join":true,"root_keys":{"app_key":{"key":"5105D918B6EEEF8A22AA92B39D9EC718"}},"mac_settings":{"rx2_data_rate_index":0,"status_time_periodicity":"0s","status_count_periodicity":0,"desired_adr_ack_limit_exponent":"ADR_ACK_LIMIT_1"},"session":{"dev_addr":"29559735","keys":{"session_key_id":"bBRMrEnueD7zAxHwX4a2RA==","f_nwk_s_int_key":{"key":"1BA000AB1BAC28B83F2A091F7EA9ABED"},"app_s_key":{"key":"927A87F8B710479D594440227BBBDD64"}},"last_f_cnt_up":10,"started_at":"2023-10-31T18:41:35.080851Z"}}

Import and check uplink (active session)

Rejoin on TTS

Regressions

NA. This is new functionality.

Notes for Reviewers

I've only tested one device for OTAA and ABP. Before merging, I will test a batch.
I'll add docs after we agree on the code.
I had one failed frame during the OTAA migration. Is this a frequency plan configuration issue? should I have used EU_863_870_TTN instead? I'm using a The Things Uno for testing.

DEBUG Failed to determine channel index of uplink {"ack": false, "adr": false, "adr_ack_req": false, "band_id": "EU_863_870", "bandwidth": 125000, "class_b": false, "data_rate": "lora:{bandwidth:125000  spreading_factor:7  coding_rate:\"4/5\"}", "dev_addr": "29559735", "error": "error:pkg/networkserver:uplink_channel_not_found (uplink channel not found)", "f_cnt_reset": false, "f_opts_len": 0, "f_port": 1, "frequency": 867700000, "full_f_cnt_up": 11, "grpc.method": "HandleUplink", "grpc.service": "ttn.lorawan.v3.GsNs", "last_f_cnt_up": 10, "m_type": "UNCONFIRMED_UP", "mac_count": 0, "mac_version": "MAC_V1_0_2", "major": "LORAWAN_R1", "namespace": "networkserver", "pending_session": false, "phy_payload_len": 14, "received_at": 1698777742.7678819, "request_id": "01HE3GPFDFN4CBZX9Q697JE83A", "spreading_factor": 7, "uplink_f_cnt": 11}

Does anything stand out to you here @adriansmares?

Checklist

Scope: The referenced issue is addressed, there are no unrelated changes.
Documentation: Relevant documentation is added or updated.
Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.

README.md

adriansmares

DEBUG Failed to determine channel index of uplink {"ack": false, "adr": false, "adr_ack_req": false, "band_id": "EU_863_870", "bandwidth": 125000, "class_b": false, "data_rate": "lora:{bandwidth:125000 spreading_factor:7 coding_rate:"4/5"}", "dev_addr": "29559735", "error": "error:pkg/networkserver:uplink_channel_not_found (uplink channel not found)", "f_cnt_reset": false, "f_opts_len": 0, "f_port": 1, "frequency": 867700000, "full_f_cnt_up": 11, "grpc.method": "HandleUplink", "grpc.service": "ttn.lorawan.v3.GsNs", "last_f_cnt_up": 10, "m_type": "UNCONFIRMED_UP", "mac_count": 0, "mac_version": "MAC_V1_0_2", "major": "LORAWAN_R1", "namespace": "networkserver", "pending_session": false, "phy_payload_len": 14, "received_at": 1698777742.7678819, "request_id": "01HE3GPFDFN4CBZX9Q697JE83A", "spreading_factor": 7, "uplink_f_cnt": 11}

This is normal to a certain extent - you don't pass any MAC state while importing the end device, which means that the LNS builds a MAC state which has the current parameters with only the 'boot' channels - 867700000 is not a 'boot' channel.

The ttnv2 source can show you how to build the MAC state and how to edit it. In this case you can do the same current = desired trick and then work back the different parameters in Firefly.

.gitignore

README.md

pkg/source/firefly/client/client.go

pkg/source/firefly/source.go

happyRip

This looks great! I like how you've streamlined the config part.

pkg/source/firefly/source.go

pkg/source/firefly/config.go

johanstokking

This JoinEUI thing is probably quite problematic, no? This only works if there's one JoinEUI per batch? And you either migrate one device, or you migrate all devices accessible with the API key?

I presume Firefly just takes the JoinEUI from the join-request, and uses the DevEUI as primary identifier?

So if there's a mix of JoinEUIs in a larger deployment, devices need to be migrated one by one?

pkg/source/firefly/client/client.go

pkg/source/firefly/config.go

pkg/source/firefly/source.go

KrishnaIyer · 2023-11-07T16:47:24Z

cmd/root.go

-			logLevel := log.InfoLevel
-			if verbose, _ := cmd.Flags().GetBool("verbose"); verbose {
-				logLevel = log.DebugLevel
-			}
-			logHandler, err := log.NewZap("console")
-			if err != nil {
-				return err
-			}
-			logger = log.NewLogger(
-				logHandler,
-				log.WithLevel(logLevel),
-			)
-			rpclog.ReplaceGrpcLogger(logger)
-			ctx = log.NewContext(ctx, logger)
-
 			exportCfg.DevIDPrefix, _ = cmd.Flags().GetString("dev-id-prefix")
-			exportCfg.EUIForID, _ = cmd.Flags().GetBool("set-eui-as-id")
-			ctx = export.NewContext(ctx, exportCfg)
-
-			cmd.SetContext(ctx)


The context was previously never propagated to the subcommands and hence this logger and the exportconfig were never used.

I've fixed the export config but we don't need this logger since we use Zap in the source.

KrishnaIyer · 2023-11-07T17:00:11Z

Retested the changes

OTAA
The device sends frame 3 on Firefly, is migrated to TTS and TTS receives frame 4 onwards correctly. Rejoin works also

ABP
The device sends frame 3 on Firefly, is migrated to TTS and TTS receives frame 4 onwards correctly

pkg/iterator/iterator_test.go

pkg/source/firefly/source.go

happyRip

Looking great

KrishnaIyer · 2023-11-08T15:04:47Z

This JoinEUI thing is probably quite problematic, no? This only works if there's one JoinEUI per batch? And you either migrate one device, or you migrate all devices accessible with the API key?

Yes indeed. The way I have documented this is to migrate devices in batches by creating a file with the Device EUIs and piping that into the devices command.
This way, all devices in the batch will have a common JoinEUI on TTS.

happyRip and others added 23 commits October 31, 2023 20:25

dev: Add firefly source

a595f8e

dev: Implement a barebones source.Source interface

edef8e7

dev: Add basic config

c82d280

dev: Improve config

0a1f175

dev: Create devices package

26f2109

dev: Add api package

326f404

dev: Extend source functionalities

cbed7ee

dev: Improve devices package

93ffebf

dev: Improve api package

cc72f7c

dev: Work on firefly package

6db8516

dev: Move to zap logger

d4e690f

dev: Add Location object to devices package

8b1f3c7

dev: Small updates

2fab5a2

doc: Add documentation for firefly source

dd978f8

cli: Fix firefly source

763c7da

all: Complete firefly functionality

7498fa0

all: Make firefly device bytes optional

938f8cd

all: Fix fetching packets

8d3ecd8

all: Update session key ID

92c815f

all: Improve code reuse

61a08e8

all: Add option to invalidate keys

6683a5f

doc: Update readme

a4a3ad8

dev: Add changelog

99603f6

KrishnaIyer added this to the Nov 2023 milestone Nov 1, 2023

KrishnaIyer self-assigned this Nov 1, 2023

KrishnaIyer commented Nov 1, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

KrishnaIyer marked this pull request as ready for review November 1, 2023 10:15

KrishnaIyer requested review from happyRip, johanstokking and adriansmares as code owners November 1, 2023 10:15

adriansmares reviewed Nov 1, 2023

View reviewed changes

happyRip reviewed Nov 1, 2023

View reviewed changes

pkg/source/firefly/source.go Outdated Show resolved Hide resolved

pkg/source/firefly/config.go Outdated Show resolved Hide resolved

johanstokking reviewed Nov 1, 2023

View reviewed changes

pkg/source/firefly/client/client.go Show resolved Hide resolved

pkg/source/firefly/config.go Outdated Show resolved Hide resolved

pkg/source/firefly/config.go Outdated Show resolved Hide resolved

pkg/source/firefly/source.go Show resolved Hide resolved

KrishnaIyer added 6 commits November 6, 2023 11:13

dev: Define iterator per source

46f8fb7

all: Harmonize logger usage

bec38d9

all: Propagate context to sub commands

0070c4c

all: Fix device ID export

609f8de

dev: Create MAC state for Firefly devices

dabeba5

all: Init flags without exposing values

9720ffa

KrishnaIyer mentioned this pull request Nov 6, 2023

Support migration from Firefly #49

Closed

3 tasks

all: Make iterator use stdin for devices

1e5e089

KrishnaIyer commented Nov 7, 2023

View reviewed changes

dev: Update readme

1b8e814

KrishnaIyer requested review from happyRip and adriansmares November 7, 2023 17:00

adriansmares reviewed Nov 8, 2023

View reviewed changes

pkg/iterator/iterator_test.go Outdated Show resolved Hide resolved

pkg/source/firefly/source.go Outdated Show resolved Hide resolved

pkg/source/firefly/source.go Outdated Show resolved Hide resolved

dev: Update test package name

012bb90

KrishnaIyer requested a review from adriansmares November 8, 2023 13:36

adriansmares approved these changes Nov 8, 2023

View reviewed changes

happyRip approved these changes Nov 8, 2023

View reviewed changes

KrishnaIyer requested a review from johanstokking November 8, 2023 15:04

KrishnaIyer merged commit d9c9b39 into master Nov 13, 2023
3 checks passed

KrishnaIyer deleted the feature/43-firefly branch November 13, 2023 11:12

KrishnaIyer mentioned this pull request Nov 13, 2023

Support migration from Firefly #43

Closed

adriansmares mentioned this pull request Mar 11, 2024

Support ChirpStack v4 #112

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support migration from the Firefly LNS #97

Support migration from the Firefly LNS #97

KrishnaIyer commented Nov 1, 2023 •

edited

Loading

adriansmares left a comment

happyRip left a comment

johanstokking left a comment

KrishnaIyer Nov 7, 2023

KrishnaIyer commented Nov 7, 2023

happyRip left a comment

KrishnaIyer commented Nov 8, 2023

Support migration from the Firefly LNS #97

Support migration from the Firefly LNS #97

Conversation

KrishnaIyer commented Nov 1, 2023 • edited Loading

Summary

Changes

Testing

Regressions

Notes for Reviewers

Checklist

adriansmares left a comment

Choose a reason for hiding this comment

happyRip left a comment

Choose a reason for hiding this comment

johanstokking left a comment

Choose a reason for hiding this comment

KrishnaIyer Nov 7, 2023

Choose a reason for hiding this comment

KrishnaIyer commented Nov 7, 2023

happyRip left a comment

Choose a reason for hiding this comment

KrishnaIyer commented Nov 8, 2023

KrishnaIyer commented Nov 1, 2023 •

edited

Loading