Skip to content

Commit

Permalink
0.3.0 (#22)
Browse files Browse the repository at this point in the history 
# Changelog

##
[0.2.1a4](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a4)
(2024-07-15)

[Full
Changelog](0.2.1a3...0.2.1a4)

**Merged pull requests:**

- Fix backwards-compat branch ref in legacy update method
[\#21](#21)
([NeonDaniel](https://github.com/NeonDaniel))

##
[0.2.1a3](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a3)
(2024-07-15)

[Full
Changelog](0.2.1a2...0.2.1a3)

##
[0.2.1a2](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a2)
(2024-06-10)

[Full
Changelog](0.2.1a1...0.2.1a2)

**Merged pull requests:**

- Add endpoint to check download status
[\#19](#19)
([NeonDaniel](https://github.com/NeonDaniel))

##
[0.2.1a1](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a1)
(2024-05-17)

[Full
Changelog](0.2.0...0.2.1a1)

**Merged pull requests:**

- Add data to update responses and fix branch bug
[\#18](#18)
([NeonDaniel](https://github.com/NeonDaniel))



\* *This Changelog was automatically generated by
[github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
  • Loading branch information
@NeonDaniel
NeonDaniel authored Jul 16, 2024
2 parents 6572624 + 9de6c41 commit 5fd8f0a
Show file tree
Hide file tree
Showing 5 changed files with 175 additions and 41 deletions.
24 changes: 18 additions & 6 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,32 @@
# Changelog

## [0.1.1a2](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.1.1a2) (2024-04-05)
## [0.2.1a4](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a4) (2024-07-15)

[Full Changelog](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/compare/0.1.1a1...0.1.1a2)
[Full Changelog](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/compare/0.2.1a3...0.2.1a4)

**Merged pull requests:**

- Update ovos-utils dependency spec [\#16](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/pull/16) ([NeonDaniel](https://github.com/NeonDaniel))
- Fix backwards-compat branch ref in legacy update method [\#21](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/pull/21) ([NeonDaniel](https://github.com/NeonDaniel))

## [0.1.1a1](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.1.1a1) (2024-03-12)
## [0.2.1a3](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a3) (2024-07-15)

[Full Changelog](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/compare/0.1.0...0.1.1a1)
[Full Changelog](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/compare/0.2.1a2...0.2.1a3)

## [0.2.1a2](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a2) (2024-06-10)

[Full Changelog](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/compare/0.2.1a1...0.2.1a2)

**Merged pull requests:**

- Add endpoint to check download status [\#19](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/pull/19) ([NeonDaniel](https://github.com/NeonDaniel))

## [0.2.1a1](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/tree/0.2.1a1) (2024-05-17)

[Full Changelog](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/compare/0.2.0...0.2.1a1)

**Merged pull requests:**

- Refactor to use neon-os Releases [\#15](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/pull/15) ([NeonDaniel](https://github.com/NeonDaniel))
- Add data to update responses and fix branch bug [\#18](https://github.com/NeonGeckoCom/neon-phal-plugin-device-updater/pull/18) ([NeonDaniel](https://github.com/NeonDaniel))



Expand Down
13 changes: 12 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ PHAL:
initramfs_url: "https://github.com/NeonGeckoCom/neon_debos/raw/{}/overlays/02-rpi4/boot/firmware/initramfs"
initramfs_path: /opt/neon/firmware/initramfs
initramfs_update_path: /opt/neon/initramfs
squashfs_url: "https://2222.us/app/files/neon_images/pi/mycroft_mark_2/updates/{}/"
squashfs_path: /opt/neon/update.squashfs
default_track: dev
```
Expand Down Expand Up @@ -54,3 +53,15 @@ Check for an available InitramFS update and emit a response with data:
```python
Message("neon.update_squashfs", {'track': 'dev'})
```

### Get Build Info
Get metadata for currently installed build:
```python
Message("neon.device_updater.get_build_info")
```

### Get Download Status
Query the plugin if an update is currently downloading:
```python
Message("neon.device_updater.get_download_status")
```
66 changes: 44 additions & 22 deletions neon_phal_plugin_device_updater/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ def __init__(self, bus=None, name="neon-phal-plugin-device-updater",
self._default_branch = self.config.get("default_track") or "master"
self._build_info = None
self._initramfs_hash = None
self._downloading = False

# Register messagebus listeners
self.bus.on("neon.check_update_initramfs", self.check_update_initramfs)
Expand All @@ -70,13 +71,15 @@ def __init__(self, bus=None, name="neon-phal-plugin-device-updater",
self.check_update_available)
self.bus.on("neon.device_updater.get_build_info",
self.get_build_info)
self.bus.on("neon.device_updater.get_download_status",
self.get_download_status)

@property
def squashfs_url(self):
log_deprecation("FTP update references are deprecated.", "1.0.0")
return self.config.get("squashfs_url", "https://2222.us/app/files/"
"neon_images/pi/mycroft_mark_2/"
"updates/{}/")
return self.config.get("squashfs_url",
"https://download.neonaiservices.com/neon_os/"
"core/rpi4/updates/{}/")

@property
def initramfs_url(self):
Expand All @@ -92,13 +95,11 @@ def initramfs_hash(self) -> Optional[str]:
if not self._initramfs_hash:
try:
Popen("mount_firmware", shell=True).wait(5)
with open(self.initramfs_real_path, "rb") as f:
self._initramfs_hash = hashlib.md5(f.read()).hexdigest()
except Exception as e:
LOG.error(e)
if isfile(self.initramfs_real_path):
with open(self.initramfs_real_path, "rb") as f:
self._initramfs_hash = hashlib.md5(f.read()).hexdigest()
if isfile(self.initramfs_real_path):
with open(self.initramfs_real_path, "rb") as f:
self._initramfs_hash = hashlib.md5(f.read()).hexdigest()
LOG.debug(f"hash={self._initramfs_hash}")
return self._initramfs_hash

Expand All @@ -124,6 +125,7 @@ def _legacy_check_initramfs_update_available(self,
@return: True if a newer initramfs is available to download
"""
branch = branch or self._default_branch
branch = "master" if branch == "stable" else branch
if not self.initramfs_url:
raise RuntimeError("No initramfs_url configured")
initramfs_url = self.initramfs_url.format(branch)
Expand Down Expand Up @@ -235,8 +237,7 @@ def _legacy_get_squashfs_latest(self, track: str = None) -> Optional[str]:

return self._stream_download_file(download_url, download_path)

@staticmethod
def _stream_download_file(download_url: str,
def _stream_download_file(self, download_url: str,
download_path: str) -> Optional[str]:
"""
Download a remote resource to a local path and return the path to the
Expand All @@ -249,6 +250,7 @@ def _stream_download_file(download_url: str,
# Download the update
LOG.info(f"Downloading update from {download_url}")
temp_dl_path = f"{download_path}.download"
self._downloading = True
try:
with requests.get(download_url, stream=True) as stream:
with open(temp_dl_path, 'wb') as f:
Expand All @@ -260,14 +262,17 @@ def _stream_download_file(download_url: str,
if file_mib < 100:
LOG.error(f"Downloaded file is too small ({file_mib}MiB)")
remove(temp_dl_path)
self._downloading = False
return
shutil.move(temp_dl_path, download_path)
LOG.info(f"Saved download to {download_path}")
self._downloading = False
return download_path
except Exception as e:
LOG.exception(e)
if isfile(temp_dl_path):
remove(temp_dl_path)
self._downloading = False

def _get_gh_latest_release_tag(self, track: str = None) -> str:
"""
Expand All @@ -280,12 +285,16 @@ def _get_gh_latest_release_tag(self, track: str = None) -> str:
valid release
"""
include_prerelease = (track or self._default_branch) in ("dev", "beta")

default_time = "2000-01-01T00:00:00Z"
url = f'https://api.github.com/repos/{self.release_repo}/releases'
releases: list = requests.get(url).json()
LOG.debug(f"Getting releases from {self.release_repo}. "
f"prerelease={include_prerelease}")
if not include_prerelease:
releases = [r for r in releases if not r.get('prerelease', True)]
url = f"{url}/latest"
release = requests.get(url).json()
return release.get("tag_name")

releases: list = requests.get(url).json()
installed_os = self.build_info.get("base_os", {}).get("name")
if not installed_os:
raise RuntimeError(f"Unable to determine installed OS from: "
Expand All @@ -309,6 +318,7 @@ def _get_gh_release_meta_from_tag(self, tag: str) -> dict:
f"{self.build_info}")
meta_url = (f"https://raw.githubusercontent.com/{self.release_repo}/"
f"{tag}/{installed_os}.yaml")
LOG.debug(f"Getting metadata from {meta_url}")
resp = requests.get(meta_url)
if not resp.ok:
raise ValueError(f"Unable to get metadata for tag={tag}")
Expand Down Expand Up @@ -355,16 +365,20 @@ def check_update_initramfs(self, message: Message):
Handle a request to check for initramfs updates
@param message: `neon.check_update_initramfs` Message
"""
branch = message.data.get("track") or self._default_branch
track = message.data.get("track") or self._default_branch
track = "beta" if track in ("dev", "beta") else "stable"
try:
meta = self._get_gh_release_meta_from_tag(
self._get_gh_latest_release_tag(self._default_branch))
self._get_gh_latest_release_tag(track))
update_available = meta['initramfs']['md5'] != self.initramfs_hash
except Exception as e:
LOG.exception(e)
update_available = self._legacy_check_initramfs_update_available(branch)
meta = dict()
update_available = self._legacy_check_initramfs_update_available(track)
self.bus.emit(message.response({"update_available": update_available,
"track": branch}))
"new_meta": meta.get('initramfs'),
"current_hash": self.initramfs_hash,
"track": track}))

def check_update_squashfs(self, message: Message):
"""
Expand Down Expand Up @@ -428,13 +442,14 @@ def update_squashfs(self, message: Message):
download_url = update_metadata['download_url'].replace(
f"/{platform}/", f"/{platform}/updates/").replace(".img.xz",
".squashfs")
download_path = join(dirname(self.initramfs_update_path),
update_metadata['build_version'])
download_path = str(join(dirname(self.initramfs_update_path),
update_metadata['build_version']))
if isfile(download_path):
LOG.info("Update already downloaded")
return download_path
update_file = self._stream_download_file(download_url,
download_path)
update_file = download_path
else:
update_file = self._stream_download_file(download_url,
download_path)
except Exception as e:
LOG.exception(f"Failed to get download_url: {e}")
update_file = self._legacy_get_squashfs_latest(track)
Expand Down Expand Up @@ -511,3 +526,10 @@ def get_build_info(self, message: Message):
@param message: `neon.device_updater.get_build_info` Message
"""
self.bus.emit(message.response(self.build_info))

def get_download_status(self, message: Message):
"""
Handle a request to check if a download is in-progress
@param message: `neon.device_updater.get_download_status` Message
"""
self.bus.emit(message.response(data={"downloading": self._downloading}))
111 changes: 100 additions & 11 deletions tests/unit_tests.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,16 @@
import logging
import unittest
from tempfile import mkstemp
from time import time
from threading import Thread
from time import time, sleep

import requests

from os import remove
from os.path import isfile, basename, join, dirname

from ovos_bus_client import Message

from neon_phal_plugin_device_updater import DeviceUpdater
from ovos_utils.messagebus import FakeBus
from ovos_utils.log import LOG
Expand Down Expand Up @@ -205,8 +209,61 @@ def test_get_gh_release_meta_from_tag(self):
self.assertTrue('/opi5/' in opi_meta['download_url'])

def test_check_update_initramfs(self):
# TODO
pass
self.plugin.initramfs_real_path = join(dirname(__file__), "initramfs")
with open(self.plugin.initramfs_real_path, 'w+') as f:
f.write("test")
self.plugin._initramfs_hash = None
self.plugin._build_info = {"base_os": {"name": "debian-neon-image-rpi4"}}

self.assertEqual(self.plugin.release_repo, "NeonGeckoCom/neon-os")

update_default = Message("neon.check_update_initramfs", {}, {})
update_stable = Message("neon.check_update_initramfs",
{"track": "stable"}, {})
update_beta = Message("neon.check_update_initramfs",
{"track": "beta"}, {})

# Test update available

# Test stable
stable_resp = self.bus.wait_for_response(update_stable).data
self.assertTrue(stable_resp['update_available'])
self.assertEqual(stable_resp['track'], "stable")
stable_meta = stable_resp['new_meta']
self.assertIsInstance(stable_meta['md5'], str)
self.assertIsInstance(stable_meta['path'], str)
self.assertEqual(stable_resp['current_hash'], self.plugin.initramfs_hash)

# Test beta
beta_resp = self.bus.wait_for_response(update_beta).data
self.assertTrue(beta_resp['update_available'])
self.assertEqual(beta_resp['track'], "beta")
beta_meta = beta_resp['new_meta']
self.assertIsInstance(beta_meta['md5'], str)
self.assertIsInstance(beta_meta['path'], str)
self.assertEqual(beta_resp['current_hash'], self.plugin.initramfs_hash)
self.assertNotEqual(stable_meta, beta_meta)

# Test default stable
self.plugin._default_branch = "master"
default_stable = self.bus.wait_for_response(update_default).data
self.assertEqual(default_stable, stable_resp)

# Test default beta
self.plugin._default_branch = "dev"
default_beta = self.bus.wait_for_response(update_default).data
self.assertEqual(default_beta, beta_resp)

# Test already updated
self.plugin._initramfs_hash = stable_meta['md5']
stable_resp = self.bus.wait_for_response(update_stable).data
self.assertFalse(stable_resp['update_available'])

self.plugin._initramfs_hash = beta_meta['md5']
beta_resp = self.bus.wait_for_response(update_beta).data
self.assertFalse(beta_resp['update_available'])

remove(self.plugin.initramfs_real_path)

def test_check_update_squashfs(self):
# TODO
Expand All @@ -221,11 +278,11 @@ def test_update_initramfs(self):
pass

def test_stream_download_file(self):
valid_os_url = "https://2222.us/app/files/neon_images/test_images/test_os.img.xz"
valid_update_file = "https://2222.us/app/files/neon_images/test_images/update_file.squashfs"
invalid_update_file = "https://2222.us/app/files/neon_images/test_images/metadata.json"
valid_path = "https://2222.us/app/files/neon_images/test_images/"
invalid_path = "https://2222.us/app/files/neon_images/invalid_directory/"
valid_os_url = "https://download.neonaiservices.com/test_images/test_os.img.xz"
valid_update_file = "https://download.neonaiservices.com/test_images/update_file.squashfs"
invalid_update_file = "https://download.neonaiservices.com/test_images/metadata.json"
valid_path = "https://download.neonaiservices.com/test_images/"
invalid_path = "https://download.neonaiservices.com/invalid_directory/"

_, output_path = mkstemp()
remove(output_path)
Expand All @@ -236,14 +293,46 @@ def test_stream_download_file(self):
self.assertFalse(isfile(output_path))
self.plugin._stream_download_file(invalid_path, output_path)
self.assertFalse(isfile(output_path))

self.plugin._stream_download_file(valid_os_url, output_path)
self.assertFalse(self.plugin._downloading)

thread = Thread(target=self.plugin._stream_download_file,
args=(valid_os_url, output_path))
thread.start()
sleep(0.5)
self.assertTrue(self.plugin._downloading)
thread.join()
self.assertTrue(isfile(output_path))
self.assertFalse(self.plugin._downloading)
remove(output_path)
self.plugin._stream_download_file(valid_update_file, output_path)

thread = Thread(target=self.plugin._stream_download_file,
args=(valid_update_file, output_path))
thread.start()
sleep(0.5)
self.assertTrue(self.plugin._downloading)
thread.join()
self.assertTrue(isfile(output_path))
self.assertFalse(self.plugin._downloading)
remove(output_path)

def test_get_build_info(self):
resp = self.plugin.bus.wait_for_response(
Message("neon.device_updater.get_build_info"))
self.assertEqual(resp.data, self.plugin.build_info)

def test_get_download_status(self):
self.assertFalse(self.plugin._downloading)
resp = self.plugin.bus.wait_for_response(Message(
"neon.device_updater.get_download_status"))
self.assertFalse(resp.data['downloading'])

self.plugin._downloading = True
resp = self.plugin.bus.wait_for_response(Message(
"neon.device_updater.get_download_status"))
self.assertTrue(resp.data['downloading'])

self.plugin._downloading = False


if __name__ == '__main__':
unittest.main()
2 changes: 1 addition & 1 deletion version.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,4 @@
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

__version__ = "0.2.0"
__version__ = "0.3.0"

0 comments on commit 5fd8f0a

Please sign in to comment.