Skip to content

Add support for splitting rollup queries #1853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 24, 2021
Merged

Add support for splitting rollup queries #1853

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
caab084
Add an SLA config flag for rollup intervals
muffix Oct 16, 2019
cdbc9d0
Add global config flag to enable splitting queries
muffix Oct 16, 2019
22b642a
Add a new SplitRollupQuery
muffix Oct 28, 2019
03e05e9
Extract an AbstractQuery
muffix Nov 11, 2019
0ec8bf9
Extract an AbstractSpanGroup
muffix Nov 11, 2019
18fab40
Avoid NullPointerException when setting start time
muffix Dec 13, 2019
6fbc98b
Scale timestamps to milliseconds for split queries
muffix Dec 13, 2019
2dd2865
Fix starting time error for split queries
muffix Dec 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fix starting time error for split queries 
Fixes a bug that would happen when the start time of a query aligns
perfectly with the time configured in the SLA for the delay of a rollup
table.
For a defined SLA, e.g. 1 day, if the start time of the query was
exactly 1 day ago, the end time of the rollups part of the query would
be updated and then be equal to its start time. That isn't allowed and
causes a query exception.
  • Loading branch information
@muffix
muffix committed Dec 30, 2019
commit 2dd2865423ed2cb8aedcd30e7412fedd57c86b82
4 changes: 2 additions & 2 deletions src/core/TsdbQuery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,8 +450,8 @@ public Deferred<Object> split(final TSQuery query, final int index, final TsdbQu

long lastRollupTimestampMillis = rollup_query.getLastRollupTimestampSeconds() * 1000L;

boolean isRawOnlyQuery = QueryUtil.isTimestampBefore(lastRollupTimestampMillis, getStartTime());
if (!isRawOnlyQuery) {
boolean needsRawAndRollupData = QueryUtil.isTimestampAfter(lastRollupTimestampMillis, getStartTime());
if (needsRawAndRollupData) {
updateRollupSplitTimes(rawQuery, lastRollupTimestampMillis);
}

Expand Down
6 changes: 3 additions & 3 deletions src/query/QueryUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,9 +409,9 @@ public static String byteRegexToString(final String regexp) {
*
* @param ts1 The first timestamp in either seconds or milliseconds.
* @param ts2 The second timestamp in either seconds or milliseconds.
* @return Whether the first timestamp is before the second
* @return Whether the first timestamp is after the second
*/
public static boolean isTimestampBefore(long ts1, long ts2) {
public static boolean isTimestampAfter(long ts1, long ts2) {
boolean ts1InSeconds = (ts1 & Const.SECOND_MASK) == 0;
boolean ts2InSeconds = (ts2 & Const.SECOND_MASK) == 0;

Expand All @@ -421,6 +421,6 @@ public static boolean isTimestampBefore(long ts1, long ts2) {
ts2 *= 1000L;
}

return ts1 < ts2;
return ts1 > ts2;
}
}
18 changes: 9 additions & 9 deletions test/query/TestQueryUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,16 +153,16 @@ public void setDataTableScanFilterEnableBoth() throws Exception {
@Test
public void timestampComparison() {
long now = DateTime.currentTimeMillis() / 1000L;
assertTrue(QueryUtil.isTimestampBefore(now*1000, now+1));
assertTrue(QueryUtil.isTimestampBefore(now-1, now*1000L));
assertTrue(QueryUtil.isTimestampBefore(now-1, now));
assertTrue(QueryUtil.isTimestampBefore((now-1)*1000L, now*1000L));
assertFalse(QueryUtil.isTimestampAfter(now*1000, now+1));
assertFalse(QueryUtil.isTimestampAfter(now-1, now*1000L));
assertFalse(QueryUtil.isTimestampAfter(now-1, now));
assertFalse(QueryUtil.isTimestampAfter((now-1)*1000L, now*1000L));

assertFalse(QueryUtil.isTimestampBefore(now+1, now*1000L));
assertFalse(QueryUtil.isTimestampBefore(now*1000L, now-1));
assertFalse(QueryUtil.isTimestampBefore(now, now-1));
assertFalse(QueryUtil.isTimestampBefore(now*1000L, (now-1)*1000L));
assertTrue(QueryUtil.isTimestampAfter(now+1, now*1000L));
assertTrue(QueryUtil.isTimestampAfter(now*1000L, now-1));
assertTrue(QueryUtil.isTimestampAfter(now, now-1));
assertTrue(QueryUtil.isTimestampAfter(now*1000L, (now-1)*1000L));

assertFalse(QueryUtil.isTimestampBefore(now, now));
assertFalse(QueryUtil.isTimestampAfter(now, now));
}
}