Logs are great. They are easy when you’re running on one or two boxes. However, they get harder when you’re running in a modest distributed environment and even more so if you’ve embraced micro-services and containers. I’ve compiled some notes and thoughts on getting Logentries running in a Kubernetes environment.
At ReactiveOps, we use Kubernetes for our clients. Centralized logs are critical for them to understand their environments and for us to assist in troubleshooting. Sometimes, a client will have existing logging tools or services they would like us to use. This is the case with my current client, who utilizes Logentries.
Getting it running in Kubernetes was not very difficult, but some things were learned. Since Kubernetes is all about containers, it makes sense that a log collector for Logentries should also run as a container. The folks at Logentries clearly feel the same way and have made the logentries/docker-logentries image available on docker hub. The instructions discuss how to run in a container, but it is limited to straight docker environment. Kubernetes is based on docker, but you do run things differently.
Getting Logentries Up and Running
I run the Logentries collector as a
DaemonSet in its own
ServiceAccount under the
kube-system namespace. The reason for running it as
DaemonSet is to ensure we run one of these containers on every node. A dedicated
ServiceAccount isn’t necessary, but helps to isolate things. You can also use a different namespace if you’d rather not add to
kube-system, but it fits pretty well since this is for logging everything.
Logentries allows you to send logs to different locations named Log Sets. In order to get logs to show up in Log Sets, you will need to create the Log Set and then an access token. Since this is a secret, we’ll store it as such. Here is an example
secret.yml to create the
logentries_token in Kubernetes:
--- apiVersion: v1 data: logentries_token: <your token | base64 -w0> kind: Secret metadata: name: logentries namespace: kube-system
Apply this with
kubectl apply -f secret.yml. Next we will create the the
serviceaccount.yml looks like this:
--- apiVersion: v1 kind: ServiceAccount metadata: name: logentries namespace: kube-system
Apply this with
kubectl apply -f serviceaccount.yml. Now we are ready for the creation of the actual work horses. The
daemonset.yml looks like this:
--- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: logentries namespace: kube-system spec: selector: matchLabels: app: logentries template: metadata: name: logentries labels: app: logentries spec: serviceAccount: logentries containers: - name: logentries image: logentries/docker-logentries imagePullPolicy: Always securityContext: privileged: true args: - "-t $LOGENTRIES_TOKEN" - "-j" # - "--no-stats" # - "--no-logs" # - "--no-dockerEvents" # - "-a <a special string>" # - "--stats-interval <STATSINTERVAL> # - "--matchByName REGEXP" # - "--matchByImage REGEXP" # - "--skipByName REGEXP" # - "--skipByImage REGEXP" volumeMounts: - name: socket mountPath: /var/run/docker.sock env: - name: LOGENTRIES_TOKEN valueFrom: secretKeyRef: name: logentries key: logentries_token volumes: - name: socket hostPath: path: /var/run/docker.sock
Which you apply with
kubectl apply -f daemonset.yml.
The above should work if you have a standard type setup, similar to the one you get from kops. If you’ve done custom installation work, you may need to adjust some things, like the location of the socket etc.
The above is the default configuration which will send container logs, container statistics, and docker events to Logentries. Should you wish to alter the logging that happens via the flags, I’ve added them in commented fashion. Details on those flags can be found on docker hub: logentries/docker-logentries/.
In general, it works as advertised. Once the
DaemonSet is running, the logs will flow. Memory utilization of each container or Pod is under 100MB and CPU consumption has been modest thus far.
The logs produced by the containers are in JSON form and the
-j switch makes sure that things pass through properly. The JSON that arrives appears clean and fully functional.
The first thing I noticed was that it can be tricky to find the logs for a single
Deployment. For example, depending on how your images or containers are named, you will likely need to use regular expressions. The simple search will not find substrings. If we have a name of
pod-staging_nginx-worker searching for
nginx, it will not return results, but
Now that we have logs for only the pod or container we’re interested in, there is the issue of all the stats that are also part of the found results. To reduce the results to only logs from the container itself, you can add another clause:
AND line. Every line logged will be stored in a JSON attribute named
, thus forcing the search to also find only those items gets us what we need:
name=/.*nginx.*/ AND line.
Another issue still in search of an answer is the use case where you’d like a Log Set for
application X and
application Y. With a classic Logentries collector, which accesses the log files directly, you can configure which Log Set the logs from a given file will go. In a Kubernetes environment, where the logs are harvested via a docker API, that is not possible. Unfortunately, Logentries does not offer a way to slice and tag once the logs are received from docker. One possible option would be to run a Logentries container alongside each pod and utilize
--no-dockerEvents. Clearly, that will lead to a large number logging containers. I will be investigating this a bit more, but the default is likely the only reasonable approach for now.
In conclusion, Logentries is perfectly usable and easy to set up with Kubernetes.