Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

Panic in function processing (seriesaggregators.go) #1939

Open
replay opened this issue Nov 5, 2020 · 8 comments
Open

Panic in function processing (seriesaggregators.go) #1939

replay opened this issue Nov 5, 2020 · 8 comments
Assignees
@fkaleo
Labels
bug
Milestone
sprint-18

Comments

@replay
Copy link
Contributor

replay commented Nov 5, 2020

In one of our production instances we're seeing a panic occurring regularly:

[Macaron] PANIC: runtime error: index out of range [179] with length 179
/usr/local/go/src/runtime/panic.go:88 (0x434fa4)
/go/src/github.com/grafana/metrictank/expr/seriesaggregators.go:208 (0xc0b96c)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:73 (0xbdcb8b)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xbdc714)
/go/src/github.com/grafana/metrictank/expr/plan.go:327 (0xc0a6a8)
/go/src/github.com/grafana/metrictank/api/graphite.go:1016 (0xc852d6)
/go/src/github.com/grafana/metrictank/api/graphite.go:318 (0xc7d424)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xb01439)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xb00e0a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xc6a124)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xc6a117)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb2cda5)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xb2cd98)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xb1f7b7)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb83d54)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xb83d3d)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)

I think this is an indication for a bigger issue, because actually this "index out of range" shouldn't happen, but as a quick fix we could at least add a len() check on the relevant line to prevent a panic.
@replay replay added the bug label Nov 5, 2020
@fkaleo fkaleo added this to the sprint-18 milestone Nov 6, 2020
@fkaleo
Copy link
Contributor

fkaleo commented Nov 6, 2020

I was able to reproduce it with query

diffSeries(summarize(groupByNode(stats.*.counters.statsd.metrics_received.count, 2, 'sum'), '1min', 'sum', false), summarize(timeShift(groupByNode(stats.*.counters.statsd.metrics_received.count, 2, 'sum'), '1d'), '1min', 'sum', false))


PANIC: runtime error: index out of range [60] with length 60

/usr/local/go/src/runtime/panic.go:88 (0x434fa4)
/go/src/github.com/grafana/metrictank/expr/seriesaggregators.go:208 (0xc0b96c)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:73 (0xbdcb8b)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xbdc714)
/go/src/github.com/grafana/metrictank/expr/plan.go:327 (0xc0a6a8)
/go/src/github.com/grafana/metrictank/api/graphite.go:1016 (0xc852d6)
/go/src/github.com/grafana/metrictank/api/graphite.go:318 (0xc7d424)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xb01439)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xb00e0a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xc6a124)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xc6a117)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb2cda5)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xb2cd98)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xb1f7b7)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb83d54)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xb83d3d)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xb01439)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xb00e0a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb85604)
/go/src/github.com/grafana/metrictank/api/middleware/tracer.go:70 (0xb855ee)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb84cce)
/go/src/github.com/grafana/metrictank/api/middleware/stats.go:76 (0xb84cb9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb2cda5)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xb2cd98)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xb1f7b7)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:187 (0xb2dff0)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:294 (0xb281cf)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:220 (0xb20b0d)
/usr/local/go/src/net/http/server.go:2843 (0x7971a2)
/usr/local/go/src/net/http/server.go:1925 (0x7929ac)
/usr/local/go/src/runtime/asm_amd64.s:1374 (0x46fde0)

@fkaleo
Copy link
Contributor

fkaleo commented Nov 6, 2020 •
edited
Loading

Even simpler reproduction query:

diffSeries(SINGLE_SERIES, summarize(timeShift(SINGLE_SERIES, '-1h'), '1min'))

Also breaks with sumSeries

@fkaleo
Copy link
Contributor

fkaleo commented Nov 6, 2020

Panic does not always happen, only with some time ranges.

@fkaleo
Copy link
Contributor

fkaleo commented Nov 7, 2020

diffSeries(summarize(metrictank.stats.docker-env.default.tank.persist.values.rate32, '1min'), summarize(timeShift(metrictank.stats.docker-env.default.tank.persist.values.count32, '-1min'), '1min')) recreates the issue.

summarize(metrictank.stats.docker-env.default.tank.persist.values.count32, '1min') produces 2 data points

summarize(timeShift(metrictank.stats.docker-env.default.tank.persist.values.count32, '-1min'), '1min') produces 3 data points

@fkaleo
Copy link
Contributor

fkaleo commented Nov 7, 2020

diffSeries(summarize(metrictank.stats.docker-env.default.tank.persist.values.rate32, '1min'), summarize(timeShift(metrictank.stats.docker-env.default.tank.persist.values.count32, '-1min'), '1min')) recreates the issue.

reversing the arguments to diffSeries and the panic seems gone

@fkaleo
Copy link
Contributor

fkaleo commented Nov 7, 2020 •
edited
Loading

Series showing the issue with from=1604708261266&to=1604708992552
image

@fkaleo
Copy link
Contributor

fkaleo commented Nov 7, 2020

in summarizeValues commenting out this piece fixes the issue:

	// MT specific bit: if !s.alignToFrom we want the output to be canonical
	// only thing needed is strip out the first point if its TS < from
	if len(out) != 0 && out[0].Ts < serie.QueryFrom {
		out = out[1:]
	}

@fkaleo
Copy link
Contributor

fkaleo commented Nov 7, 2020

very linked to previously fixed issue #1811

@fkaleo fkaleo mentioned this issue Nov 11, 2020
Closed
@fkaleo fkaleo self-assigned this Nov 11, 2020
@Dieterbe Dieterbe removed their assignment Sep 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@fkaleo fkaleo

Labels
bug
Projects
None yet
Milestone
sprint-18
Development

Successfully merging a pull request may close this issue.

timeShift function: adequately set QueryFrom/QueryTo on output series grafana/metrictank
3 participants
@Dieterbe @replay @fkaleo