add fillmode "last" to sql datasource

This adds a new fill mode last (last observation carried forward) for grafana
to the sql datasources. This fill mode will fill in the last seen value in a
series when a timepoint is missing or NULL if no value for that series has
been seen yet.

docs/sources/features/datasources/mssql.md

@@ -81,7 +81,9 @@ Macro example | Description
 *$__timeFrom()* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'*
 *$__timeTo()* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'*
 *$__timeGroup(dateColumn,'5m'[, fillvalue])* | Will be replaced by an expression usable in GROUP BY clause. Providing a *fillValue* of *NULL* or *floating value* will automatically fill empty series in timerange with that value. <br/>For example, *CAST(ROUND(DATEDIFF(second, '1970-01-01', time_column)/300.0, 0) as bigint)\*300*.
-*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so all null values will be converted to the fill value (all null values would be set to zero using this example).
+*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
+*$__timeGroup(dateColumn,'5m', NULL)* | Same as above but NULL will be used as value for missing points.
+*$__timeGroup(dateColumn,'5m', last)* | Same as above but the last seen value in that series will be used as fill value if no value has been seen yet NULL will be used.
 *$__timeGroupAlias(dateColumn,'5m')* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
 *$__unixEpochFilter(dateColumn)* | Will be replaced by a time range filter using the specified column name with times represented as unix timestamp. For example, *dateColumn > 1494410783 AND dateColumn < 1494497183*
 *$__unixEpochFrom()* | Will be replaced by the start of the currently active time selection as unix timestamp. For example, *1494410783*

docs/sources/features/datasources/mysql.md

@@ -64,7 +64,9 @@ Macro example | Description
 *$__timeFrom()* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'*
 *$__timeTo()* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'*
 *$__timeGroup(dateColumn,'5m')* | Will be replaced by an expression usable in GROUP BY clause. For example, *cast(cast(UNIX_TIMESTAMP(dateColumn)/(300) as signed)*300 as signed),*
-*$__timeGroup(dateColumn,'5m',0)* | Same as above but with a fill parameter so all null values will be converted to the fill value (all null values would be set to zero using this example).
+*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
+*$__timeGroup(dateColumn,'5m', NULL)* | Same as above but NULL will be used as value for missing points.
+*$__timeGroup(dateColumn,'5m', last)* | Same as above but the last seen value in that series will be used as fill value if no value has been seen yet NULL will be used.
 *$__timeGroupAlias(dateColumn,'5m')* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
 *$__unixEpochFilter(dateColumn)* | Will be replaced by a time range filter using the specified column name with times represented as unix timestamp. For example, *dateColumn > 1494410783 AND dateColumn < 1494497183*
 *$__unixEpochFrom()* | Will be replaced by the start of the currently active time selection as unix timestamp. For example, *1494410783*

docs/sources/features/datasources/postgres.md

@@ -61,7 +61,9 @@ Macro example | Description
 *$__timeFrom()* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'*
 *$__timeTo()* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'*
 *$__timeGroup(dateColumn,'5m')* | Will be replaced by an expression usable in GROUP BY clause. For example, *(extract(epoch from dateColumn)/300)::bigint*300*
-*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so all null values will be converted to the fill value (all null values would be set to zero using this example).
+*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
+*$__timeGroup(dateColumn,'5m', NULL)* | Same as above but NULL will be used as value for missing points.
+*$__timeGroup(dateColumn,'5m', last)* | Same as above but the last seen value in that series will be used as fill value if no value has been seen yet NULL will be used.
 *$__timeGroupAlias(dateColumn,'5m')* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
 *$__unixEpochFilter(dateColumn)* | Will be replaced by a time range filter using the specified column name with times represented as unix timestamp. For example, *dateColumn >= 1494410783 AND dateColumn <= 1494497183*
 *$__unixEpochFrom()* | Will be replaced by the start of the currently active time selection as unix timestamp. For example, *1494410783*

pkg/tsdb/mssql/macros.go

@@ -99,9 +99,13 @@ func (m *msSqlMacroEngine) evaluateMacro(name string, args []string) (string, er
 		if len(args) == 3 {
 			m.query.Model.Set("fill", true)
 			m.query.Model.Set("fillInterval", interval.Seconds())
-			if args[2] == "NULL" {
-				m.query.Model.Set("fillNull", true)
-			} else {
+			switch args[2] {
+			case "NULL":
+				m.query.Model.Set("fillMode", "null")
+			case "last":
+				m.query.Model.Set("fillMode", "last")
+			default:
+				m.query.Model.Set("fillMode", "value")
 				floatVal, err := strconv.ParseFloat(args[2], 64)
 				if err != nil {
 					return "", fmt.Errorf("error parsing fill value %v", args[2])

pkg/tsdb/mssql/macros_test.go

@@ -76,12 +76,25 @@ func TestMacroEngine(t *testing.T) {
 				_, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', NULL)")
 
 				fill := query.Model.Get("fill").MustBool()
-				fillNull := query.Model.Get("fillNull").MustBool()
+				fillMode := query.Model.Get("fillMode").MustString()
 				fillInterval := query.Model.Get("fillInterval").MustInt()
 
 				So(err, ShouldBeNil)
 				So(fill, ShouldBeTrue)
-				So(fillNull, ShouldBeTrue)
+				So(fillMode, ShouldEqual, "null")
+				So(fillInterval, ShouldEqual, 5*time.Minute.Seconds())
+			})
+
+			Convey("interpolate __timeGroup function with fill (value = last)", func() {
+				_, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', last)")
+
+				fill := query.Model.Get("fill").MustBool()
+				fillMode := query.Model.Get("fillMode").MustString()
+				fillInterval := query.Model.Get("fillInterval").MustInt()
+
+				So(err, ShouldBeNil)
+				So(fill, ShouldBeTrue)
+				So(fillMode, ShouldEqual, "last")
 				So(fillInterval, ShouldEqual, 5*time.Minute.Seconds())
 			})
 

pkg/tsdb/mysql/macros.go

@@ -94,9 +94,13 @@ func (m *mySqlMacroEngine) evaluateMacro(name string, args []string) (string, er
 		if len(args) == 3 {
 			m.query.Model.Set("fill", true)
 			m.query.Model.Set("fillInterval", interval.Seconds())
-			if args[2] == "NULL" {
-				m.query.Model.Set("fillNull", true)
-			} else {
+			switch args[2] {
+			case "NULL":
+				m.query.Model.Set("fillMode", "null")
+			case "last":
+				m.query.Model.Set("fillMode", "last")
+			default:
+				m.query.Model.Set("fillMode", "value")
 				floatVal, err := strconv.ParseFloat(args[2], 64)
 				if err != nil {
 					return "", fmt.Errorf("error parsing fill value %v", args[2])

pkg/tsdb/mysql/mysql_test.go

@@ -295,7 +295,7 @@ func TestMySQL(t *testing.T) {
 
 			})
 
-			Convey("When doing a metric query using timeGroup with float fill enabled", func() {
+			Convey("When doing a metric query using timeGroup with value fill enabled", func() {
 				query := &tsdb.TsdbQuery{
 					Queries: []*tsdb.Query{
 						{
@@ -320,6 +320,35 @@ func TestMySQL(t *testing.T) {
 				points := queryResult.Series[0].Points
 				So(points[3][0].Float64, ShouldEqual, 1.5)
 			})
+
+			Convey("When doing a metric query using timeGroup with last fill enabled", func() {
+				query := &tsdb.TsdbQuery{
+					Queries: []*tsdb.Query{
+						{
+							Model: simplejson.NewFromAny(map[string]interface{}{
+								"rawSql": "SELECT $__timeGroup(time, '5m', last) as time_sec, avg(value) as value FROM metric GROUP BY 1 ORDER BY 1",
+								"format": "time_series",
+							}),
+							RefId: "A",
+						},
+					},
+					TimeRange: &tsdb.TimeRange{
+						From: fmt.Sprintf("%v", fromStart.Unix()*1000),
+						To:   fmt.Sprintf("%v", fromStart.Add(34*time.Minute).Unix()*1000),
+					},
+				}
+
+				resp, err := endpoint.Query(nil, nil, query)
+				So(err, ShouldBeNil)
+				queryResult := resp.Results["A"]
+				So(queryResult.Error, ShouldBeNil)
+
+				points := queryResult.Series[0].Points
+				So(points[2][0].Float64, ShouldEqual, 15.0)
+				So(points[3][0].Float64, ShouldEqual, 15.0)
+				So(points[6][0].Float64, ShouldEqual, 20.0)
+			})
+
 		})
 
 		Convey("Given a table with metrics having multiple values and measurements", func() {

pkg/tsdb/postgres/macros.go

@@ -116,9 +116,13 @@ func (m *postgresMacroEngine) evaluateMacro(name string, args []string) (string,
 		if len(args) == 3 {
 			m.query.Model.Set("fill", true)
 			m.query.Model.Set("fillInterval", interval.Seconds())
-			if args[2] == "NULL" {
-				m.query.Model.Set("fillNull", true)
-			} else {
+			switch args[2] {
+			case "NULL":
+				m.query.Model.Set("fillMode", "null")
+			case "last":
+				m.query.Model.Set("fillMode", "last")
+			default:
+				m.query.Model.Set("fillMode", "value")
 				floatVal, err := strconv.ParseFloat(args[2], 64)
 				if err != nil {
 					return "", fmt.Errorf("error parsing fill value %v", args[2])

pkg/tsdb/postgres/postgres_test.go

@@ -276,7 +276,7 @@ func TestPostgres(t *testing.T) {
 
 			})
 
-			Convey("When doing a metric query using timeGroup with float fill enabled", func() {
+			Convey("When doing a metric query using timeGroup with value fill enabled", func() {
 				query := &tsdb.TsdbQuery{
 					Queries: []*tsdb.Query{
 						{
@@ -303,6 +303,34 @@ func TestPostgres(t *testing.T) {
 			})
 		})
 
+		Convey("When doing a metric query using timeGroup with last fill enabled", func() {
+			query := &tsdb.TsdbQuery{
+				Queries: []*tsdb.Query{
+					{
+						Model: simplejson.NewFromAny(map[string]interface{}{
+							"rawSql": "SELECT $__timeGroup(time, '5m', last), avg(value) as value FROM metric GROUP BY 1 ORDER BY 1",
+							"format": "time_series",
+						}),
+						RefId: "A",
+					},
+				},
+				TimeRange: &tsdb.TimeRange{
+					From: fmt.Sprintf("%v", fromStart.Unix()*1000),
+					To:   fmt.Sprintf("%v", fromStart.Add(34*time.Minute).Unix()*1000),
+				},
+			}
+
+			resp, err := endpoint.Query(nil, nil, query)
+			So(err, ShouldBeNil)
+			queryResult := resp.Results["A"]
+			So(queryResult.Error, ShouldBeNil)
+
+			points := queryResult.Series[0].Points
+			So(points[2][0].Float64, ShouldEqual, 15.0)
+			So(points[3][0].Float64, ShouldEqual, 15.0)
+			So(points[6][0].Float64, ShouldEqual, 20.0)
+		})
+
 		Convey("Given a table with metrics having multiple values and measurements", func() {
 			type metric_values struct {
 				Time                time.Time

pkg/tsdb/sql_engine.go

@@ -274,9 +274,15 @@ func (e *sqlQueryEndpoint) transformToTimeSeries(query *Query, rows *core.Rows,
 	fillMissing := query.Model.Get("fill").MustBool(false)
 	var fillInterval float64
 	fillValue := null.Float{}
+	fillLast := false
+
 	if fillMissing {
 		fillInterval = query.Model.Get("fillInterval").MustFloat64() * 1000
-		if !query.Model.Get("fillNull").MustBool(false) {
+		switch query.Model.Get("fillMode").MustString() {
+		case "null":
+		case "last":
+			fillLast = true
+		case "value":
 			fillValue.Float64 = query.Model.Get("fillValue").MustFloat64()
 			fillValue.Valid = true
 		}
@@ -352,6 +358,14 @@ func (e *sqlQueryEndpoint) transformToTimeSeries(query *Query, rows *core.Rows,
 					intervalStart = series.Points[len(series.Points)-1][1].Float64 + fillInterval
 				}
 
+				if fillLast {
+					if len(series.Points) > 0 {
+						fillValue = series.Points[len(series.Points)-1][0]
+					} else {
+						fillValue.Valid = false
+					}
+				}
+
 				// align interval start
 				intervalStart = math.Floor(intervalStart/fillInterval) * fillInterval
 
@@ -377,6 +391,14 @@ func (e *sqlQueryEndpoint) transformToTimeSeries(query *Query, rows *core.Rows,
 			intervalStart := series.Points[len(series.Points)-1][1].Float64
 			intervalEnd := float64(tsdbQuery.TimeRange.MustGetTo().UnixNano() / 1e6)
 
+			if fillLast {
+				if len(series.Points) > 0 {
+					fillValue = series.Points[len(series.Points)-1][0]
+				} else {
+					fillValue.Valid = false
+				}
+			}
+
 			// align interval start
 			intervalStart = math.Floor(intervalStart/fillInterval) * fillInterval
 			for i := intervalStart + fillInterval; i < intervalEnd; i += fillInterval {

public/app/plugins/datasource/mssql/partials/query.editor.html

@@ -53,7 +53,9 @@ Macros:
 - $__timeEpoch(column) -> DATEDIFF(second, '1970-01-01', column) AS time
 - $__timeFilter(column) -> column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z'
 - $__unixEpochFilter(column) -> column >= 1492750877 AND column <= 1492750877
-- $__timeGroup(column, '5m'[, fillvalue]) -&gt; CAST(ROUND(DATEDIFF(second, '1970-01-01', column)/300.0, 0) as bigint)*300. Providing a <i>fillValue</i> of <i>NULL</i> or floating value will automatically fill empty series in timerange with that value.
+- $__timeGroup(column, '5m'[, fillvalue]) -&gt; CAST(ROUND(DATEDIFF(second, '1970-01-01', column)/300.0, 0) as bigint)*300.
+     by setting fillvalue grafana will fill in missing values according to the interval
+     fillvalue can be either a literal value, NULL or last; last will fill in the last seen value or NULL if none has been seen yet
 - $__timeGroupAlias(column, '5m'[, fillvalue]) -&gt; CAST(ROUND(DATEDIFF(second, '1970-01-01', column)/300.0, 0) as bigint)*300 AS [time]
 
 Example of group by and order by with $__timeGroup:

public/app/plugins/datasource/mysql/partials/query.editor.html

@@ -53,7 +53,9 @@ Macros:
 - $__timeEpoch(column) -> UNIX_TIMESTAMP(column) as time_sec
 - $__timeFilter(column) -> column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z'
 - $__unixEpochFilter(column) ->  time_unix_epoch > 1492750877 AND time_unix_epoch < 1492750877
-- $__timeGroup(column,'5m') -> cast(cast(UNIX_TIMESTAMP(column)/(300) as signed)*300 as signed)
+- $__timeGroup(column,'5m'[, fillvalue]) -> cast(cast(UNIX_TIMESTAMP(column)/(300) as signed)*300 as signed)
+     by setting fillvalue grafana will fill in missing values according to the interval
+     fillvalue can be either a literal value, NULL or last; last will fill in the last seen value or NULL if none has been seen yet
 - $__timeGroupAlias(column,'5m') -> cast(cast(UNIX_TIMESTAMP(column)/(300) as signed)*300 as signed) AS "time"
 
 Example of group by and order by with $__timeGroup:

public/app/plugins/datasource/postgres/partials/query.editor.html

@@ -53,7 +53,9 @@ Macros:
 - $__timeEpoch -> extract(epoch from column) as "time"
 - $__timeFilter(column) -> column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z'
 - $__unixEpochFilter(column) ->  column >= 1492750877 AND column <= 1492750877
-- $__timeGroup(column,'5m') -> (extract(epoch from column)/300)::bigint*300
+- $__timeGroup(column,'5m'[, fillvalue]) -> (extract(epoch from column)/300)::bigint*300
+     by setting fillvalue grafana will fill in missing values according to the interval
+     fillvalue can be either a literal value, NULL or last; last will fill in the last seen value or NULL if none has been seen yet
 - $__timeGroupAlias(column,'5m') -> (extract(epoch from column)/300)::bigint*300 AS "time"
 
 Example of group by and order by with $__timeGroup: