Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codes index #12

Merged
merged 25 commits into from
Feb 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
5b7f86d
initial index-support
asura6 May 9, 2023
f56a6f2
Merge branch 'codes-index' into initial-index-support
Quba1 Jan 27, 2024
31c7cde
Merge pull request #2 from asura6/initial-index-support
Quba1 Jan 27, 2024
5df6d99
make index_handle public only within crate
Quba1 May 17, 2023
a3c25cb
re-export CodesIndex in lib.rs for consistency
Quba1 Jan 27, 2024
e76f3bc
make codes_index a feature
Quba1 Jan 27, 2024
0e1ff8e
mark featured modules in docs
Quba1 Jan 27, 2024
4b0847e
reolve cargo doc warnings
Quba1 Jan 27, 2024
95c59b6
update gh actions
Quba1 Jan 27, 2024
993efb9
fix testing in gh actions
Quba1 Jan 27, 2024
db3555c
(deosn't compile) extract index binds to separate file
Quba1 Jan 28, 2024
8909556
WIP mostly finish CodesIndex API and conversion to CH
Quba1 Jan 28, 2024
640122f
WIP codes_handle iterator with index works
Quba1 Jan 28, 2024
5a30252
compiles but does not allow for editing index
Quba1 Jan 28, 2024
952a111
actually, the API makes sense (and tests now pass)
Quba1 Jan 28, 2024
2590cb2
move high-level index tests to integration tests
Quba1 Jan 29, 2024
9f8a137
add integration tests with multithreading
Quba1 Jan 29, 2024
bb2558c
add mutex for problematic functions
Quba1 Jan 30, 2024
9d8d83a
add error and panic tests for codes_index
Quba1 Jan 30, 2024
c099d58
add features to gh actions
Quba1 Jan 30, 2024
cff3157
add index handle interference test
Quba1 Jan 30, 2024
d3682ff
clippy
Quba1 Jan 30, 2024
166d701
make ec_index default feature
Quba1 Feb 3, 2024
58ab75e
safeguard all index functions
Quba1 Feb 3, 2024
21a624e
make index feature experimental
Quba1 Feb 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/rust-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ jobs:
cargo clean
- name: Test with cargo
run: |
RUST_BACKTRACE=full cargo test
RUST_BACKTRACE=full cargo test --features "experimental_index"
- name: Check with clippy
run: |
cargo clippy -- -W clippy::pedantic
cargo clippy --features "experimental_index"
- name: Build release
run: |
cargo build --release
cargo build --release --features "experimental_index"

build-macos:

Expand All @@ -45,10 +45,10 @@ jobs:
cargo clean
- name: Test with cargo
run: |
RUST_BACKTRACE=full cargo test
RUST_BACKTRACE=full cargo test --features "experimental_index"
- name: Check with clippy
run: |
cargo clippy -- -W clippy::pedantic
cargo clippy --features "experimental_index"
- name: Build release
run: |
cargo build --release
cargo build --release --features "experimental_index"
8 changes: 4 additions & 4 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ jobs:
cargo clean
- name: Build with cargo
run: |
cargo build --release
cargo build --release --features "experimental_index"
cargo clean
- name: Test with cargo
run: |
cargo test
cargo test --features "experimental_index"
cargo clean
- name: Benchmark with criterion
run: |
Expand All @@ -53,11 +53,11 @@ jobs:
cargo clean
- name: Build with cargo
run: |
cargo build --release
cargo build --release --features "experimental_index"
cargo clean
- name: Test with cargo
run: |
cargo test
cargo test --features "experimental_index"
cargo clean
- name: Benchmark with criterion
run: |
Expand Down
9 changes: 5 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name = "eccodes"
description = "Unofficial high-level Rust bindings of the latest ecCodes release"
repository = "https://github.com/ScaleWeather/eccodes"
version = "0.8.0"
version = "0.9.0"
readme = "README.md"
authors = ["Jakub Lewandowski <[email protected]>"]
keywords = ["eccodes", "grib", "bufr", "meteorology", "weather"]
Expand All @@ -18,7 +18,7 @@ edition = "2021"
exclude = [".github/*", ".vscode/*", ".idea/*", "data/*"]

[dependencies]
eccodes-sys = "0.5.1"
eccodes-sys = "0.5.2"
libc = "0.2"
thiserror = "1.0"
bytes = "1.5"
Expand All @@ -29,17 +29,18 @@ num-traits = "0.2"
fallible-iterator = "0.3"

[dev-dependencies]
eccodes-sys = "0.5.1"
reqwest = { version = "0.11", features = ["rustls-tls"] }
tokio = { version = "1.35", features = ["macros", "rt"] }
criterion = "0.5"
testing_logger = "0.1"
rand = "0.8"

[features]
docs = ["eccodes-sys/docs"]
experimental_index = []

[package.metadata.docs.rs]
features = ["docs"]
features = ["docs", "experimental_index"]

[[bench]]
name = "main"
Expand Down
Binary file added data/iceland-surface.idx
Binary file not shown.
49 changes: 41 additions & 8 deletions src/codes_handle/iterator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ use crate::{
codes_handle_new_from_message_copy,
},
};
#[cfg(feature = "experimental_index")]
use crate::{intermediate_bindings::codes_index::codes_handle_new_from_index, CodesIndex};

use super::GribFile;

///`FallibleIterator` implementation for `CodesHandle` to access GRIB messages inside file.
///
Expand All @@ -18,7 +22,7 @@ use crate::{
///Therefore this crate utilizes the `Iterator` to provide the access to GRIB messages in
///a safe and convienient way.
///
///[`FallibleIterator`](fallible_iterator::FallibleIterator) is used instead of classic `Iterator`
///[`FallibleIterator`] is used instead of classic `Iterator`
///because internal ecCodes functions can return error codes when the GRIB file
///is corrupted and for some other reasons. The usage of `FallibleIterator` is sligthly different
///than usage of `Iterator`, check its documentation for more details.
Expand Down Expand Up @@ -77,23 +81,52 @@ use crate::{
///## Errors
///The `next()` method will return [`CodesInternal`](crate::errors::CodesInternal)
///when internal ecCodes function returns non-zero code.
impl FallibleIterator for CodesHandle {
impl FallibleIterator for CodesHandle<GribFile> {
type Item = KeyedMessage;

type Error = CodesError;

fn next(&mut self) -> Result<Option<Self::Item>, Self::Error> {
let new_eccodes_handle;
unsafe {
codes_handle_delete(self.eccodes_handle)?;
new_eccodes_handle = codes_handle_new_from_file(self.source.pointer, self.product_kind);
}

match new_eccodes_handle {
Ok(h) => {
self.eccodes_handle = h;

if self.eccodes_handle.is_null() {
Ok(None)
} else {
let message = get_message_from_handle(h);
Ok(Some(message))
}
}
Err(e) => Err(e),
}
}
}

#[cfg(feature = "experimental_index")]
impl FallibleIterator for CodesHandle<CodesIndex> {
type Item = KeyedMessage;

type Error = CodesError;

fn next(&mut self) -> Result<Option<Self::Item>, Self::Error> {
let file_handle;
let new_eccodes_handle;
unsafe {
codes_handle_delete(self.file_handle)?;
file_handle = codes_handle_new_from_file(self.file_pointer, self.product_kind);
codes_handle_delete(self.eccodes_handle)?;
new_eccodes_handle = codes_handle_new_from_index(self.source.pointer);
}

match file_handle {
match new_eccodes_handle {
Ok(h) => {
self.file_handle = h;
self.eccodes_handle = h;

if self.file_handle.is_null() {
if self.eccodes_handle.is_null() {
Ok(None)
} else {
let message = get_message_from_handle(h);
Expand Down
2 changes: 1 addition & 1 deletion src/codes_handle/keyed_message/iterator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ use super::KeysIteratorFlags;
///so it is probably more efficient to call that function directly only for keys you
///are interested in.
///
///[`FallibleIterator`](fallible_iterator::FallibleIterator) is used instead of classic `Iterator`
///[`FallibleIterator`] is used instead of classic `Iterator`
///because internal ecCodes functions can return internal error in some edge-cases.
///The usage of `FallibleIterator` is sligthly different than usage of `Iterator`,
///check its documentation for more details.
Expand Down
2 changes: 1 addition & 1 deletion src/codes_handle/keyed_message/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ impl Drop for KeyedMessage {
///Technical note: delete functions in ecCodes can only fail with [`CodesInternalError`](crate::errors::CodesInternal::CodesInternalError)
///when other functions corrupt the inner memory of pointer, in that case memory leak is possible.
///In case of corrupt pointer segmentation fault will occur.
///The pointers are cleared at the end of drop as they ar not not functional despite the result of delete functions.
///The pointers are cleared at the end of drop as they are not functional despite the result of delete functions.
fn drop(&mut self) {
if let Some(nrst) = self.nearest_handle {
unsafe {
Expand Down
Loading
Loading