In this guide, we will closely look at Kubernetes Events and the intricacies of the kubectl get events command. We will talk about everything from the types of Kubernetes events to how to use kubectl get events with real-world examples.
Applications don't always run in a steady state, as you might expect. User activity varies depending on the day, the hour, or certain occasions. Runtime needs for memory, networking, and CPU fluctuate.
It's crucial to react to these occurrences effectively. However, it's as crucial to comprehend how and why particular apps are behaving in this way.
Let's investigate how Kubernetes event objects are used to provide some context.
- What are Kubectl Events - The Basics
- Where do Kubernetes Events Get Stored?
- Types of Kubectl Events - Kubernetes Event Classifications
- How to Use Kubectl Get Events (with Examples)
- How to Filter & Monitor Kubectl Events
- Frequently Asked Questions
What are Kubectl Events - The Basics
In Kubernetes, Kubectl Events provide eventful insights into the state of your cluster, helping you monitor and troubleshoot its health and performance.
Think of them as real-time notifications that you get on your phone, here, they inform you about various occurrences within your Kubernetes environment.
These events are emitted by Kubernetes components, including controllers, schedulers, and other system-level entities.
Kubectl Events covers a wide range of activities, such as pod creations, deletions, errors, node issues, and more.
They act as an invaluable source of information for developers to understand what's happening under the hood and to identify potential problems before they escalate within a certain time frame.
Yes, these events do not stay for eternity but for one hour post the occurrence of the event.
Also Read: What are Kubernetes Operators?
Where do Kubernetes Events Get Stored?
Kubernetes events are API objects that get stored in the apiserver on master in your cluster.
Now that you know what Kubernetes events are, it will make more sense to understand their various types.
Also Read: Docker vs Containerd
Types of Kubectl Events - Kubernetes Event Classifications
Kubernetes events are a helpful tool for monitoring your system and catching potential problems. By keeping track of these events and setting up alerts, you can quickly detect and resolve issues.
These events can be broadly classified into five types. Familiarizing yourself with these five types of events will help you understand event tracking better.
1. Failed Events
These kubectl events occur when there's an error with your object manifest file or a problem pulling a container image from the repository.
To troubleshoot this, check the image address and tags using the command: `docker pull <image_name>`.
Common causes point towards incorrect credentials for private repositories, rate limiting, or possible typos in the image name or tags.
2. Evicted Events
If a node in your Kubernetes cluster lacks sufficient resources, which is not that rare, it may evict pods with resource constraints.
But, you can prevent evictions by using taints to stop scheduling new pods on heavily utilized nodes.
3. Failed Scheduling Events
These events happen when there aren't enough nodes available to schedule a new pod. Kinda a cranky situation!
To diagnose the issue, use: `kubectl describe object/object-name`.
Also Read: Kubernetes Node vs Pod vs Cluster
4. Volume Events
As we know persistent volumes are crucial for data storage in Kubernetes workloads, if you encounter "FailedMount" or "FailedAttachVolume" events, it could be due to networking or configuration errors between persistent volumes and claims.
To debug, use `kubectl describe pv` and `kubectl describe pvc` to get more details about the volumes and claims.
5. Node Events
Nodes are the machines in your cluster running applications and storing data but, unhealthy nodes can lead to various issues, like 5XX errors or problems with deployments.
To troubleshoot this, you can try rebooting the node manually or use the control plane to do it automatically.
Let's look at some other types of events/states in Kubernetes.
1. CrashLoopBackOff - This takes place whenever a pod starts, crashes, restarts, and crashes once more.
2. ImagePullBackOff - When the node is unable to obtain the picture.
3. Evicted Events - These can occur when a node decides to terminate or evict a pod in order to free up resources (CPU, memory, etc.). K8s is supposed to reschedule the pod on another node if this occurs.
4. FailedMount / FailedAttachVolume - this event stops pods from starting if the persistent volume or storage they need cannot be accessed.
5. FailedSchedulingEvents - When the scheduler is unable to locate a node on which to run your pods.
6. NodeNotReady - when a node is unable to launch a pod due to a more fundamental problem.
How to Use Kubectl Get Events (with Examples)
The `kubectl get events` command is your window into the world of Kubernetes events. You can use it to fetch valuable insights into your cluster's activities.
Here are some examples of using `kubectl get events`.
1. Displaying All Events
To view all events occurring within your Kubernetes cluster, use the following command:
kubectl get events
2. Filtering Events by Namespace
If you want to focus on events from a specific namespace, use the `--namespace` flag:
kubectl get events --namespace mynamespace
3. Watching Events in Real-time
For real-time monitoring of events, you can use the `watch` flag with the 'kubectl events' command:
kubectl get events --watch
4. Getting Events for a Specific Object
To retrieve events related to a specific object, such as a pod or deployment, use the object's name and specify its type:
kubectl get events --field-selector involvedObject.name=my-pod --field-selector involvedObject.kind=Pod
5. Filtering Events by Type
As mentioned earlier, you can filter events based on their types (Normal, Warning, or Error) using the `--field-selector` flag:
kubectl get events --field-selector type=Warning
Also Read: A Complete Guide to Kubectl Proxy
How to Access Kubernetes Events?
By default, Kubernetes events are stored in the cluster and can be accessed using the following command:
kubectl get events
Here's a sample output of the command.
Explanation of the columns:
LAST SEEN: The time elapsed since the Kubernetes event was last seen or recorded.
TYPE: The type of the event, which can be Normal, Warning, or Error.
REASON: The reason for the event, indicating the nature of the event.
OBJECT: The object associated with the event, such as a pod or a deployment.
MESSAGE: A short description or message providing more details about the event.
In the sample output:
1. A pod named `nginx-deployment-abc123` was successfully scheduled on `node-1`.
2. The pod is currently in the process of pulling the latest `nginx` image from the repository.
3. The image was pulled successfully in approximately 1.23 seconds, and the container `nginx` was created.
4. The `nginx` container in the pod has started successfully.
5. There's a warning indicating that the `nginx-deployment-abc123` pod is experiencing back-off restarting due to a failed container.
6. Another warning event indicates that the pod `nginx-deployment-xyz456` failed to schedule because no nodes were available that tolerate its taints.
If you are not using one of the popular monitoring tools, you can use kubectl or access the API directly using a script. Simply pass the -o json flag to any obtain operation to get JSON output from kubectl.
kubectl get events -o json
How to Filter & Monitor Kubectl Events?
Our `kubectl get events` command can produce a lot of information, especially in big clusters. To make it more manageable and to make logical output, we can filter events using the `--field-selector` flag.
For instance, if we only want to see events related to pods, we can use this command:
kubectl get events --field-selector involvedObject.kind=Pod -n <namespace>
What this will do is it will display events where the involvedObject is a pod. You can also apply other selectors to further narrow down events based on criteria like reason, type, and source.
You can create your own exporter to export Kubernetes events if Prometheus is configured to monitor your cluster. When events occur, you can then set up notifications in Prometheus using AlertManager.
By setting the DD_COLLECT_KUBERNETES_EVENTS environment variable for your agents to true, you can enable Kubernetes event collection if you use Datadog.
After that, you can set up monitors for the event-related purposes of your choice.
Moreover, the eventhandler_config is a Grafana Agent Kubernetes integration that lets your agent retrieve real-time events from the Kubernetes API and transmit them to Grafana Loki, which serves as your log aggregator.
How to Filter Kubernetes Events Based on Timestamps?
Use kubectl get events --sort-by='.metadata.creationTimestamp' -A to sort according to the creation time of each namespace.
To sort out the occurrence of events by “last seen” timestamp, use the below command:
kubectl get events --sort-by=’.lastTimestamp’
Frequently Asked Questions
1. How can I see what's happening in Kubernetes?
To see what's happening in Kubernetes, you can use the command `kubectl get events`. It will show you a list of recent activities in your cluster.
2. How can I view logs for events in a Kubernetes pod?
To view logs for events in a Kubernetes pod, you can use the command `kubectl describe pod`. It will give you detailed information about the pod and any related events.
3. What's the command to see events for a specific pod?
To see events for a specific pod, use the command `kubectl get events --field-selector involvedObject.name=my-pod --field-selector involvedObject.kind=Pod`. Just replace `my-pod` with the name of your pod.
4. How can I keep an eye on Kubernetes events?
To keep an eye on Kubernetes events in real-time, use the command `kubectl get events --watch`. It will continuously show you new events as they happen in your cluster.
5. How do I check events for the whole Kubernetes cluster?
To check events for the entire Kubernetes cluster, use the command `kubectl get events`. By default, it shows events from the entire cluster, giving you an overview of its activities. If you want to focus on specific namespaces or objects, you can add more options to the command using the `--namespace` flag as shown earlier.