ENH: Using msgpack instead of json #1819

* ENH: New expert bot for uWhoisd * ENH: Code cleanup, improve tests * DOC: Add documentation for uWhoisd

Referring to my mail I created a Parser& Bot which is able to Collect and parse data from fireeye mail and file analysis appliances. We are collection sha1 and MD5 hashed and if there is network communication we are also collecting urls and domains.

rewrite bot bots' parameter handling to current methodology use HTTP mixin for fireeye collector

Fireeye has more products than the MAS system

adds the module name of bots as section anchor adds the module name to the information block if it was missing or incomplete fixes part of certtools/intelmq-api#4

As some RDAP implementation may vary, we check if the entity is a valid entity-type. As of the RFC for RDAP only JSON dicts are allowed. Fixes #1942 Signed-off-by: Sebastian Waldbauer <[email protected]>

- adapted docs - added news entry - added upgrade config - added upgrade config test - added html table parser test fixes #1938

... and add legacy tag to drone_brute_force which it replaces

add legacy tag to application_ddos_victim Co-authored-by: Wagner <[email protected]>

add documentation for #1950 and #1952 fix filename detection for the new feeds

The cache mixin replaces the cache lib and adds the possibility to inherit the mixin in classes. Signed-off-by: Sebastian Waldbauer <[email protected]>

Signed-off-by: Sebastian Waldbauer <[email protected]>

This commit adds license information to a lot of files and adds a .reuse/dep5 file that lists the license information for some folders The commit also changes the main license in setup.cfg from AGPL-3.0-only to AGPL-3.0-or-later because only one file has the AGPL-3.0-only file as license and multiple files have the AGPL-3.0-or-later in the license header. It also removes the cef_logo.png file, as there is no information about the licese anywhere to be found. It is now included directly from the website of the european union. Closes #1633

and add legacy tag to shadowserver caida config

and add legacy tag to darknet config

and add legacy tag to the configs it replaces and update changelog and documentation accordingly

Closes #1972

See https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-exchange-server-report/

fix mapping use compromised type if the data indicates an active webshell plus add testcases add changelog update bots documentation

enhance mappings add 4/6 agnostic mapping for `Sinkhole-Events` as well document feeds with IPv4 and IPv6 better and shorter

This commit adds a license header or a license file to most of the files, or documents the license in the .reuse/dep5 license file. Some of the process was automated, first by listing all the files that are not reuse lint compliant: > reuse lint > ../reuse.lst This list was then modified to remove metainformation and only list filenames. Also a couple of filenames that need manual modification were removed. Then using git and reuse: > for file in `cat ../reuse.lst`; do year=`git log --reverse --pretty="format:%ai" $file | head -1 | cut -d "-" -f 1`; author=`git log --reverse --pretty="format:%an" $file|head -1`; reuse addheader --copyright="$author" --year="$year" --license="AGPL-3.0-or-later" --skip-unrecognised $file; done Then the same process was repeated for files reuse does not recognize, like csv and json files or REQUIREMENTS.txt files.

match with RSIT in the taxonomy intrusions: compromised -> system-compromise unauthorized-command -> system-compromise unauthorized-login -> system-compromise adapt bots depending on the name add changelog and news entries, including SQL update statements

merged into information-content-security > unauthorised-information-modification adapt bots depending on the name add changelog and news entries, including SQL update statements

was renamed and marked as deprecated in 2.0.0.beta1 #1404

Compatibility with the deprecated configuration format (before 1.0.0.dev7) was removed. #1404

The deprecated shell scripts - `update-asn-data` - `update-geoip-data` - `update-tor-nodes` - `update-rfiprisk-data` have been removed in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in `contrib/cron-jobs/intelmq-update-database`. #1404

add two n6 images directly to the repository, as they are not displayed on readthedocs otherwise: The other websites hosting the images block loading images if the referer does not match a whitelist. we can't add a noreferer HTML attribute in rst as well. the option left is to add the files, that only implies adding the licensing information and the AGPL-3.0 license text as well. add two illustrations on the the flow n6 to intelmq and vice versa, own work. some textual improvements in the document itself.

The Aggregate Expert might be used to aggregate events within a given timespan and threshold. Signed-off-by: Sebastian Waldbauer <[email protected]>

Using msgpack instead of json results in faster (de)serialize and less memory usage. Redis is also capable of msgpack within its lua api i.e. https://github.com/kengonakajima/lua-msgpack-native. ====== Benchmark ======= JSON median size: 387 MSGPACK median size: 329 ------------------------ Diff: 16.20% JSON * Serialize: 39286 * Deserialize: 30713 MSGPACK * Serialize: 23483 * Deserialize: 12602 --------------------- DIFF * Serialize: 50.35% * Deserialize: 83.62% Data extracted from spamhaus-collector Measurements based on deduplicator-expert 460 events in total process by deducplicator-expert Signed-off-by: Sebastian Waldbauer <[email protected]>

Signed-off-by: Sebastian Waldbauer <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Using msgpack instead of json #1819

ENH: Using msgpack instead of json #1819

Commits on Jul 15, 2022

ENH: Using msgpack instead of json #1819

Are you sure you want to change the base?

ENH: Using msgpack instead of json #1819

Commits on Jul 15, 2022