If you’re looking to monitor resource usage at the pod level in Kubernetes, you have a few solid options depending on how deep you want to go — from basic CPU/memory usage to detailed application metrics.

1. Metrics Server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

kubectl top pod -n your-namespace

url : https://192.168.122.121:10250/metrics/cadvisor Unauthorized

Why /metrics on 10250 Says “Unauthorized”

• Port 10250 is Kubelet’s HTTPS endpoint
• It requires TLS client certificates, a valid ServiceAccount, or Bearer token
• Browsing directly via browser (or curl) without a token will show Unauthorized

FIX

Step 1: Create a ServiceAccount with Permissions

 apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: prometheus-remote
   namespace: kube-system
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
   name: prometheus-kubelet
 rules:
   - apiGroups: [""]
     resources:
       - nodes/proxy
       - nodes/stats
       - nodes/metrics
     verbs: ["get", "list", "watch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
   name: prometheus-kubelet-binding
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: prometheus-kubelet
 subjects:
   - kind: ServiceAccount
     name: prometheus-remote
     namespace: kube-system

     kubectl apply -f prometheus-sa.yaml

Step 2: Extract the Bearer Token

 SECRET=$(kubectl -n kube-system get sa prometheus-remote -o jsonpath="{.secrets[0].name}")
 kubectl -n kube-system get secret $SECRET -o jsonpath="{.data.token}" | base64 -d > /etc/prometheus/kubelet.token

got blank

Fix: Manually create a token Secret Apply this YAML to create a token tied to the prometheus-remote ServiceAccount:

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-remote-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: prometheus-remote
type: kubernetes.io/service-account-token

kubectl apply -f prometheus-remote-token.yaml

Wait a few seconds for the token to be populated

kubectl -n kube-system get secret prometheus-remote-token -o jsonpath="{.data.token}" | base64 -d > /etc/prometheus/kubelet.token

Step 3: Edit prometheus.yml on the External Prometheus Server

scrape_configs:
  - job_name: 'kubelet'
    scheme: https
    metrics_path: /metrics
    bearer_token_file: /etc/prometheus/kubelet.token
    tls_config:
      insecure_skip_verify: true  # Accept self-signed certs (or configure proper CA)
    static_configs:
      - targets:
          - 192.168.122.121:10250  # Replace with your node's IP

To scrape container stats (like CPU/mem per pod), add:

- job_name: 'kubelet-cadvisor'
  scheme: https
  metrics_path: /metrics/cadvisor
  bearer_token_file: /etc/prometheus/kubelet.token
  tls_config:
    insecure_skip_verify: true
  static_configs:
    - targets:
        - 192.168.122.121:10250