Signed-off-by: yangxuan <[email protected]>

pr: milvus-io#29675

Signed-off-by: aoiasd <[email protected]>

…o#29858)

See also milvus-io#29803

This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index

---------

Signed-off-by: Congqi Xia <[email protected]>

this supports mmap for marisa trie index
related milvus-io#21866

Signed-off-by: yah01 <[email protected]>

…9860)

milvus-io#29759

Signed-off-by: luzhang <[email protected]>
Co-authored-by: luzhang <[email protected]>

issue: milvus-io#29793 
Use `DocSetCollector` instead of `TopDocsCollector`, which will avoid
scoring and sorting.

---------

Signed-off-by: longjiquan <[email protected]>

this protect the loading index from failure, and speed up the loading
progress

Signed-off-by: yah01 <[email protected]>

If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed

---------

Signed-off-by: Wei Liu <[email protected]>

issue: milvus-io#29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager

---------

Signed-off-by: Wei Liu <[email protected]>

…9873)

This pull request simplifies the integration test for cross-cluster
routing by reusing `integration.MiniClusterSuite`, instead of defining
custom Milvus clients, servers, and etcd client.

issue: milvus-io#29874

Signed-off-by: bigsheeper <[email protected]>

…#29870)

this leads to the EntitiesNum metric would be never reduced

fix: milvus-io#29766

Signed-off-by: yah01 <[email protected]>

issue：milvus-io#22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <[email protected]>

milvus-io#29759

Signed-off-by: luzhang <[email protected]>
Co-authored-by: luzhang <[email protected]>

This PR change some frequency log to rated level

Signed-off-by: Wei Liu <[email protected]>

Signed-off-by: Sammy Huang <[email protected]>

Signed-off-by: Sammy Huang <[email protected]>

…9641)

milvus-io#22882
/kind feature

Signed-off-by: wayblink <[email protected]>

Signed-off-by: nico <[email protected]>

pr: milvus-io#29634 

After verifying that the problem is fixed, it is necessary to run this
case every night for regression.

Signed-off-by: elstic <[email protected]>

Update Pytest image changes
See changes:
milvus-io@930735c
Signed-off-by: sre-ci-robot [email protected]

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

issue: milvus-io#29846

Signed-off-by: jaime <[email protected]>

…e up (milvus-io#29890)

milvus-io#29889

Signed-off-by: Sammy Huang <[email protected]>

Update Knowhere Commit
Signed-off-by: sre-ci-robot [email protected]

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Add tracing span for search/query operation waiting tsafe duration

Signed-off-by: Congqi Xia <[email protected]>

milvus-io#29912)

related milvus-io#29843

Signed-off-by: yah01 <[email protected]>

as we have the pool size limit so we don't need to limit the concurrency
manually

Signed-off-by: yah01 <[email protected]>

Delete detail log will be large and hard to read when log level is
debug. This PR change the log to stringer and print only pk range,
number.

Signed-off-by: Congqi Xia <[email protected]>

Signed-off-by: Sammy Huang <[email protected]>

issue: milvus-io#29781
issue: milvus-io/milvus-proto#237

Signed-off-by: zhenshan.cao <[email protected]>

this utilizes concurrent loading

Signed-off-by: yah01 <[email protected]>

issue: milvus-io#29335

---------

Signed-off-by: sunby <[email protected]>

issue: milvus-io#29814
if channel is not subscribed yet, the generated load segment task will
be remove from task scheduler due to the load segment task need to be
transfer to worker node by shard leader.

This PR skip generate load segment task when channel is not subscribed
yet.

Signed-off-by: Wei Liu <[email protected]>

issue: milvus-io#29897

---------

Signed-off-by: Cai Zhang <[email protected]>

Related issue: milvus-io#29883 
xfail for now.

Signed-off-by: yanliang567 <[email protected]>

…milvus-io#29945)

See also: milvus-io#29657

Datanode Compactor use estimated row number from schema to decide when
to sync the batch of data when executing compaction. This est value
could go way from actual size when the schema contains variable field(
say VarChar, JSON, etc.)

This PR make compactor able to check the actual buffer data size and
make it possible to sync when buffer is actually beyong max binglog
size.

---------

Signed-off-by: Congqi Xia <[email protected]>

…-io#29922)

See also: milvus-io#29650

Either segment dml position & channel checkpoint could be newer in some
cases. This PR make PackLoadSegments use the newer one improving load
performance during cases where there are lots of upsert.

---------

Signed-off-by: Congqi Xia <[email protected]>

Signed-off-by: lixinguo <[email protected]>
Co-authored-by: lixinguo <[email protected]>

…o#29948)

issue: milvus-io#29766
also see pr: milvus-io#29825

Signed-off-by: chyezh <[email protected]>

issue: milvus-io#29846

Since checking if Milvus running inside the container is difficult,
which is decided by the system version and cgroup version, we try to get
the memory limit of the container each time instead of checking whether
inside the container first.

Signed-off-by: jaime <[email protected]>

/kind improvement

Signed-off-by: SimFG <[email protected]>

…rt pointer protect. (milvus-io#29932)

may related issue: milvus-io#29828

Signed-off-by: chyezh <[email protected]>

…us-io#29848)

issue: milvus-io#29818

Signed-off-by: chyezh <[email protected]>

Signed-off-by: lixinguo <[email protected]>
Co-authored-by: lixinguo <[email protected]>

This fix will not augment the execution time of unit tests, but solely
enhances tolerance for waiting for failure.

issue: milvus-io#29921

Signed-off-by: bigsheeper <[email protected]>

when apply dynamic config changes, we should format the value to proper
unit
This PR fix update rate limit config with wrong value.

Signed-off-by: Wei Liu <[email protected]>

See also: milvus-io#29696

Signed-off-by: yangxuan <[email protected]>

related: milvus-io#29844

Signed-off-by: MrPresent-Han <[email protected]>

issue: milvus-io#29973

Signed-off-by: sunby <[email protected]>

according to benchmark, the GPU pool size with 6 performs best

Signed-off-by: yah01 <[email protected]>

…lvus-io#30014)

Ignoring .git may result "unusable report" in codecov. Use checkout
action to get .git information in upload codecov step

---------

Signed-off-by: Congqi Xia <[email protected]>

the recent changes move the level 0 segments list to a new proto field,
which leads to the QueryCoord can't see the level 0 segments, handle the
new changes
fix milvus-io#29907

Signed-off-by: yah01 <[email protected]>

…o#29974)

issue: milvus-io#30025

Signed-off-by: Sammy Huang <[email protected]>

milvus-io#29845)

related: milvus-io#29844

Signed-off-by: MrPresent-Han <[email protected]>

…-io#30020)

See also milvus-io#28817

Signed-off-by: Congqi Xia <[email protected]>

…30022)

See also milvus-io#29935

Signed-off-by: Congqi Xia <[email protected]>

Signed-off-by: yangxuan <[email protected]>

Signed-off-by: yangxuan <[email protected]>

…9999)

issue: milvus-io#29677 milvus-io#29838
during get shard leaders, if qeurynode doesn't ack the heartbeat than
10s, querycoord will treat it as unavailable, and won't return shard
leader on it. but when querynode has a full cpu usage, it's easily to
stuck for more than 10s without ack the heartbeat, which cause no shard
leader to search/query.

This PR remove heartbeat lag logic during get shard leaders

Signed-off-by: Wei Liu <[email protected]>

this converts the segcore error to merr if possible

Signed-off-by: yah01 <[email protected]>

this avoids a corner case: after load index failed, this index can be
never loaded as it has been added into the segment's index map

Signed-off-by: yah01 <[email protected]>

`C.CString` result needs to be freed after usage

Signed-off-by: Congqi Xia <[email protected]>

…#30063)

Signed-off-by: Sammy Huang <[email protected]>

…30047)

issue: milvus-io#30046

to workaround the error "HOME not set"  happened in self hosted runner

solution from
https://alexanderhose.com/aws-codecommit-with-instance-role-and-user-data/

Signed-off-by: Sammy Huang <[email protected]>

…io#29832)

this also checks the param value
fix milvus-io#29909

Signed-off-by: yah01 <[email protected]>

…us-io#30057)

we sum the total data size in int32, which could lead to an overflow
error
related milvus-io#30056

Signed-off-by: yah01 <[email protected]>

…0062)

Some important request log level shall be at least Info level

Signed-off-by: Congqi Xia <[email protected]>

issue : milvus-io#30000

Signed-off-by: zhenshan.cao <[email protected]>

issue: milvus-io#29841
if segment loaded, submit load segment task for it isn't permitted, to
avoid load segment twice. but this logic blocks the leader checker to
correct leader view by `LoadSegment`

This PR remove the segment loaded check, to fix that leader checker
cann't submit load task

Signed-off-by: Wei Liu <[email protected]>

… run failed instead abandon (milvus-io#29621)

Signed-off-by: aoiasd <[email protected]>

Signed-off-by: Congqi Xia <[email protected]>

issue: milvus-io#29841
This PR fix leader checker use wrong check interval, which causes leader
checker trigger too frequency

Signed-off-by: Wei Liu <[email protected]>

Signed-off-by: sre-ci-robot <[email protected]>

issue: milvus-io#29791 
/kind improvement
Signed-off-by: xige-16 <[email protected]>

Signed-off-by: xige-16 <[email protected]>

issue: milvus-io#29840 milvus-io#29867
/kind bug

Signed-off-by: xige-16 <[email protected]>

Signed-off-by: xige-16 <[email protected]>

…d rate limit (milvus-io#30061)

fix milvus-io#30060

Signed-off-by: yah01 <[email protected]>

…0073)

currently integration test may timeout if any case run time is above 3
minutes. This duration was hard coded.

This PR change this duration into a customized parameter and could be
passed via test running commands.

---------

Signed-off-by: Congqi Xia <[email protected]>

issue: milvus-io#30000
related to: [milvus-proto
milvus-io#202](milvus-io/milvus-proto#202)

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <[email protected]>

fix vulnerability alerts

Signed-off-by: zhuwenxing <[email protected]>

issue: milvus-io#30074
This PR fix that management restful api in proxy doesn't register to
http service

Signed-off-by: Wei Liu <[email protected]>

don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
milvus-io#28885

Signed-off-by: lixinguo <[email protected]>
Co-authored-by: lixinguo <[email protected]>

…ilvus-io#30105)

/kind improvement

Signed-off-by: SimFG <[email protected]>

This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode

See also: milvus-io#30099

Signed-off-by: yangxuan <[email protected]>

…29979)

See also: milvus-io#25323, milvus-io#29969
many users reported log file name is incorrect when starting in mixture
type.

---------

Signed-off-by: sunby <[email protected]>

this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve milvus-io#30095

Signed-off-by: yah01 <[email protected]>

milvus-io#30127)

See also: milvus-io#28873

When datanode returns error or go offline during GetCompactionResult
call, the compress binlog logic will panic since it was using a nil
result

This PR move it after the CheckRPCCall error to prevent this case.

Signed-off-by: Congqi Xia <[email protected]>

Signed-off-by: aoiasd <[email protected]>

…ilvus-io#30122)

See also: milvus-io#30121 milvus-io#27675

This PR changes the delete buffering logic:
- Write buffer shall buffer insert first
- Then the delete messages shall be evaluated
  - Whether PK matches previous Bloom filter, which ts is always smaller
  - Whether PK matches insert data which has smaller timestamp
- Then the segment bloom filter is updates by the newly buffered pk rows

---------

Signed-off-by: Congqi Xia <[email protected]>

issue: milvus-io#29791
/kind improvement

Signed-off-by: xige-16 <[email protected]>

---------

Signed-off-by: xige-16 <[email protected]>

See also: milvus-io#29204

Signed-off-by: yangxuan <[email protected]>

issue: milvus-io#28499

Signed-off-by: nico <[email protected]>

milvus-io#23075

Signed-off-by: nico <[email protected]>

issue: milvus-io#29943

---------

Signed-off-by: sunby <[email protected]>

issue: milvus-io#30142

Signed-off-by: bigsheeper <[email protected]>

…30145)

Resolves: milvus-io#30144

Signed-off-by: Congqi Xia <[email protected]>