|
@@ -67,9 +67,9 @@ The autocomplete menu can be trigger by pressing Ctrl + Space. The Autocomplete
|
|
|
|
|
|
|
|
Suggestions can appear under the query field - click on them to update your query with the suggested change.
|
|
Suggestions can appear under the query field - click on them to update your query with the suggested change.
|
|
|
|
|
|
|
|
-- For counters (monotonously increasing metrics), a rate function will be suggested.
|
|
|
|
|
-- For buckets, a histogram function will be suggested.
|
|
|
|
|
-- For recording rules, possible to expand the rules.
|
|
|
|
|
|
|
+* For counters (monotonously increasing metrics), a rate function will be suggested.
|
|
|
|
|
+* For buckets, a histogram function will be suggested.
|
|
|
|
|
+* For recording rules, possible to expand the rules.
|
|
|
|
|
|
|
|
### Table Filters
|
|
### Table Filters
|
|
|
|
|
|
|
@@ -79,6 +79,8 @@ Click on the filter button <span title="Filter for label" class="logs-label__ico
|
|
|
|
|
|
|
|
For Grafana 6.0, the first log integration is for the new open source log aggregation system from Grafana Labs - [Loki](https://github.com/grafana/loki). Loki is designed to be very cost effective, as it does not index the contents of the logs, but rather a set of labels for each log stream. The logs from Loki are queried in a similar way to querying with label selectors in Prometheus. It uses labels to group log streams which can be made to match up with your Prometheus labels. Read more about Grafana Loki [here](https://github.com/grafana/loki) or the Grafana Labs hosted variant: [Grafana Cloud Logs](https://grafana.com/loki).
|
|
For Grafana 6.0, the first log integration is for the new open source log aggregation system from Grafana Labs - [Loki](https://github.com/grafana/loki). Loki is designed to be very cost effective, as it does not index the contents of the logs, but rather a set of labels for each log stream. The logs from Loki are queried in a similar way to querying with label selectors in Prometheus. It uses labels to group log streams which can be made to match up with your Prometheus labels. Read more about Grafana Loki [here](https://github.com/grafana/loki) or the Grafana Labs hosted variant: [Grafana Cloud Logs](https://grafana.com/loki).
|
|
|
|
|
|
|
|
|
|
+See the [Loki's data source documentation](../datasources/loki) on how to query for log data.
|
|
|
|
|
+
|
|
|
### Switching from Metrics to Logs
|
|
### Switching from Metrics to Logs
|
|
|
|
|
|
|
|
If you switch from a Prometheus query to a logs query (you can do a split first to have your metrics and logs side by side) then it will keep the labels from your query that exist in the logs and use those to query the log streams. For example, the following Prometheus query:
|
|
If you switch from a Prometheus query to a logs query (you can do a split first to have your metrics and logs side by side) then it will keep the labels from your query that exist in the logs and use those to query the log streams. For example, the following Prometheus query:
|
|
@@ -91,67 +93,18 @@ after switching to the Logs datasource, the query changes to:
|
|
|
|
|
|
|
|
This will return a chunk of logs in the selected time range that can be grepped/text searched.
|
|
This will return a chunk of logs in the selected time range that can be grepped/text searched.
|
|
|
|
|
|
|
|
-### Log Queries
|
|
|
|
|
-
|
|
|
|
|
-A log query consists of two parts: **log stream selector**, and a **search expression**. For performance reasons you need to start by choosing a log stream by selecting a log label.
|
|
|
|
|
-
|
|
|
|
|
-The Logs Explorer (the `Log labels` button) next to the query field shows a list of labels of available log streams. An alternative way to write a query is to use the query field's autocomplete - you start by typing a left curly brace `{` and the autocomplete menu will suggest a list of labels. Press the `enter` key to execute the query.
|
|
|
|
|
-
|
|
|
|
|
-Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.
|
|
|
|
|
-
|
|
|
|
|
-<div class="medium-6 columns">
|
|
|
|
|
- <video width="800" height="500" controls>
|
|
|
|
|
- <source src="/assets/videos/explore_loki.mp4" type="video/mp4">
|
|
|
|
|
- Your browser does not support the video tag.
|
|
|
|
|
- </video>
|
|
|
|
|
-</div>
|
|
|
|
|
-
|
|
|
|
|
-<br />
|
|
|
|
|
-
|
|
|
|
|
-#### Log Stream Selector
|
|
|
|
|
-
|
|
|
|
|
-For the label part of the query expression, wrap it in curly braces `{}` and then use the key value syntax for selecting labels. Multiple label expressions are separated by a comma:
|
|
|
|
|
-
|
|
|
|
|
-`{app="mysql",name="mysql-backup"}`
|
|
|
|
|
-
|
|
|
|
|
-The following label matching operators are currently supported:
|
|
|
|
|
-
|
|
|
|
|
-- `=` exactly equal.
|
|
|
|
|
-- `!=` not equal.
|
|
|
|
|
-- `=~` regex-match.
|
|
|
|
|
-- `!~` do not regex-match.
|
|
|
|
|
-
|
|
|
|
|
-Examples:
|
|
|
|
|
-
|
|
|
|
|
-- `{name=~"mysql.+"}`
|
|
|
|
|
-- `{name!~"mysql.+"}`
|
|
|
|
|
-
|
|
|
|
|
-The [same rules that apply for Prometheus Label Selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#instant-vector-selectors) apply for Loki Log Stream Selectors.
|
|
|
|
|
-
|
|
|
|
|
-Another way to add a label selector, is in the table section, clicking on the **Filter** button beside a label will add the label to the query expression. This even works for multiple queries and will the label selector to each query.
|
|
|
|
|
-
|
|
|
|
|
-#### Search Expression
|
|
|
|
|
-
|
|
|
|
|
-After writing the Log Stream Selector, you can filter the results further by writing a search expression. The search expression can be just text or a regex expression.
|
|
|
|
|
-
|
|
|
|
|
-Example queries:
|
|
|
|
|
-
|
|
|
|
|
-- `{job="mysql"} error`
|
|
|
|
|
-- `{name="kafka"} tsdb-ops.*io:2003`
|
|
|
|
|
-- `{instance=~"kafka-[23]",name="kafka"} kafka.server:type=ReplicaManager`
|
|
|
|
|
-
|
|
|
|
|
### Deduping
|
|
### Deduping
|
|
|
|
|
|
|
|
Log data can be very repetitive and Explore can help by hiding duplicate log lines. There are a few different deduplication algorithms that you can use:
|
|
Log data can be very repetitive and Explore can help by hiding duplicate log lines. There are a few different deduplication algorithms that you can use:
|
|
|
|
|
|
|
|
-- `exact` Exact matches are done on the whole line, except for date fields.
|
|
|
|
|
-- `numbers` Matches on the line after stripping out numbers (durations, IP addresses etc.).
|
|
|
|
|
-- `signature` The most aggressive deduping - strips all letters and numbers, and matches on the remaining whitespace and punctuation.
|
|
|
|
|
|
|
+* `exact` Exact matches are done on the whole line, except for date fields.
|
|
|
|
|
+* `numbers` Matches on the line after stripping out numbers (durations, IP addresses etc.).
|
|
|
|
|
+* `signature` The most aggressive deduping - strips all letters and numbers, and matches on the remaining whitespace and punctuation.
|
|
|
|
|
|
|
|
### Timestamp, Local time and Labels
|
|
### Timestamp, Local time and Labels
|
|
|
|
|
|
|
|
There are some other check boxes under the logging graph apart from the Deduping options.
|
|
There are some other check boxes under the logging graph apart from the Deduping options.
|
|
|
|
|
|
|
|
-- Timestamp: shows/hides the Timestamp column
|
|
|
|
|
-- Local time: shows/hides the Local time column
|
|
|
|
|
-- Labels: shows/hides the label filters column
|
|
|
|
|
|
|
+* Timestamp: shows/hides the Timestamp column
|
|
|
|
|
+* Local time: shows/hides the Local time column
|
|
|
|
|
+* Labels: shows/hides the label filters column
|