The following guide provides detailed troubleshooting steps for GreyNoise Sensors that are unhealthy or not functioning as expected. Before diving into the sections below, we recommend trying the quickest path to resolution first: replacing or reinstalling the sensor.

Quick Resolution: Reinstall or Replace

If your sensor is unhealthy and you want the fastest path back to a working state, try one of these options before attempting deeper troubleshooting.

Option 1 (Recommended): Reinstall the Sensor Software

The fastest way to recover is to uninstall and reinstall the sensor software on the existing host. Before reinstalling, disable the old sensor in the GreyNoise Visualizer — since the new sensor will register with the same IP address, the old registration must be removed first.

📘 Note

In most cases, uninstalling and reinstalling is sufficient. However, the uninstall script can't account for everything — for example, if other packages were installed on the host that conflict with sensor operation. If the uninstall fails or you're unable to successfully reinstall after it completes, try Option 2.

Option 2: Replace the Host

A slower but more reliable approach is to replace the host entirely with a fresh one and perform a clean install. If the new host will use the same IP as the old one, disable the old sensor in the GreyNoise Visualizer before installing on the new host, otherwise the IP addresses will conflict and registration will fail.

Identifying the Problem

If you'd prefer to troubleshoot rather than replace, start by checking the sensor's detail page in the GreyNoise Visualizer. Navigate to Observe → Sensors, then click View on the affected sensor.

Under the Status section, you will see one or more reasons indicating why the sensor is unhealthy. Each reason corresponds to a section in this guide. Some reasons may include a more detailed list of specific components that are unhealthy, displayed at the bottom of the status box.

Prerequisites

The troubleshooting sections below assume familiarity with:

• Navigating the Linux filesystem from the command line
• Reading and modifying files from the command line
• YAML syntax
• apt package management
• systemd — specifically systemctl for managing services and journalctl for reading logs

Some sections may require familiarity with additional tools, which will be noted.

Missing Health Monitoring Agent

This status means the Sensor Agent (greygent) is not installed on your sensor. Without it, GreyNoise cannot collect detailed health information about the sensor or determine what might be broken on the host.

First step: Install the Sensor Agent to enable detailed health monitoring. Once installed and running, it will automatically report any health issues it finds — and those issues may be the root cause of why your sensor is unhealthy.

If you believe you already installed the Sensor Agent, verify by running:

apt show greygent

If you see No packages found, the agent is not installed. If you see package information displayed, the agent is installed but likely did not start successfully. See the Sensor Agent Troubleshooting section below.

Sensor Agent Troubleshooting

Check the Sensor Agent logs by running:

journalctl -xeu greygent

If you see errors related to loading the config, parsing the workspace, or constructing the greygent context, there is likely a configuration issue.

The configuration file is located at /opt/greynoise/greygent.yml. At a minimum, it should contain the following:

sensor_service_url: https://api.greynoise.io
sensor_service_token: <YOUR_API_TOKEN>
workspace: <YOUR_WORKSPACE_ID>
health_check_interval_seconds: 300
nat: false

Make sure the correct values are in place for <YOUR_API_TOKEN> and <YOUR_WORKSPACE_ID>. You can retrieve these from the GreyNoise Visualizer by clicking your username in the top-right corner, then selecting API Key.

🚧

Workspace ID Format

<YOUR_WORKSPACE_ID> is the UUID for your workspace, not the workspace name.

If you provided any additional flags during the initial sensor installation (such as -t for NAT), you may also see a bootstrap_options section in the configuration file. Note that the -t/--nat flag corresponds to the top-level nat field — if you provided that flag during install, you should see nat: true.

📘

When to use NAT mode

NAT mode (nat: true) should be enabled when the sensor host sits behind a NAT gateway — meaning the host's local IP address differs from the public IP address that GreyNoise sees. When enabled, the sensor configures DNAT rules on the Wireguard interface to correctly translate between the host's local address and the public sensor IP. If you are unsure whether your host is behind NAT, compare the output of ip addr on the host with the public IP shown in the GreyNoise Visualizer — if they differ, you need NAT mode.

Once you have verified the configuration file, restart the agent:

systemctl restart greygent

If issues persist or you see repeated errors in the logs, try updating the Sensor Agent to the latest version — the issue may already have been fixed. If problems continue after updating, contact [email protected].

Unsupported Sensor Host Environment

This means the operating system or Linux kernel version on the host is not supported. There is no way to repair this on the existing host — you will need to replace the sensor using Option 2 above with a host running a supported environment.

Supported environments:

• Linux Kernel >= 5.6
• Ubuntu 22.04 or later LTS release

See the Sensor Installation Guide for full system requirements.

Sensor Configuration Issue

The Sensor Agent monitors a number of host-level components that require specific configuration for proper sensor operation. If any component is misconfigured — whether by accident, interference from other software, or drift — it will be surfaced as a configuration issue.

You can check the Sensor Agent service status and logs with:

systemctl status greygent
journalctl -xeu greygent

The Sensor Agent runs a check cycle every 5 minutes by default. During each cycle, it generates log statements for every component it checks. Healthy components produce an INFO log entry; unhealthy components produce ERROR logs with details about what's wrong. In some cases, you will also see logs indicating the agent is attempting to auto-repair the issue.

The sections below correspond to the specific component identifiers you'll see on the sensor detail page and in the agent logs.

Stale Last Seen (sensor_stale_last_seen)

This means the Sensor Agent has not recently reported health status back to GreyNoise. The agent may have crashed, or may be encountering errors that prevent it from reporting. See the Sensor Agent Troubleshooting section above for steps to diagnose and restart the agent.

IPTables (sensor_iptables)

❗

Caution

This section assumes familiarity with iptables. Misconfiguring packet filtering rules could lock you out of your sensor host.

GreyNoise Sensors require specific packet filtering rules in iptables. If rules are missing, extra, or misconfigured, the sensor may not operate properly.

The Sensor Agent can auto-repair these issues. In most cases, the problem will be fixed on the next check cycle (every 5 minutes). If the issue persists across multiple cycles, you can try fully resetting iptables and letting the agent rebuild from a clean state.

First, ensure the filter table accepts all packets:

iptables -t filter -P INPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT
iptables -t filter -P OUTPUT ACCEPT

Then flush all rules and custom chains from each table. For example, to clear the filter table:

sudo iptables -t filter -F
sudo iptables -t filter -X

Once complete, only the default chains should remain for each table and each chain should be empty. The Sensor Agent should recover on the next cycle.

If the problem still persists, check the agent logs to see if auto-repairs are running successfully. There may be another package or service on the host that is interfering (such as NetworkManager or another firewall manager). Disabling and uninstalling any such software is necessary for the sensor to stay healthy. If you're unsure what might be causing interference, consider replacing the host with a fresh one per Option 2.

Route Rules (sensor_route_rules)

❗

Caution

This section assumes familiarity with the ip command for managing routing rules. Misconfiguring routing rules could lock you out of your sensor host.

Be very careful with ip rule flush! Running this command on its own will lock you out of the host. If you must flush all rules, do so in a script that immediately adds rules back to preserve SSH access.

GreyNoise Sensors use custom routing rules to ensure packets are correctly routed between the internet, the sensor, and the GreyNoise platform.

The Sensor Agent can auto-repair these issues. If the issue persists across multiple cycles, you can repair the rules manually.

The sensor expects the following routing rules (where $INTERFACE is the name of the network interface that connects to the internet):

100:    from all fwmark 0x3 iif wg0 lookup 4
101:    from all fwmark 0x3 iif $INTERFACE lookup 3
200:    from all lookup local
32766:  from all lookup main
32767:  from all lookup default

If these rules are missing, add them with ip rule add. Then delete any extra rules that shouldn't be present with ip rule del.

📘

Important

The local routing rule must be at priority 200, with priorities 100 and 101 before it. If this ordering is incorrect, the sensor will not work even if all other configuration is correct.

Route Tables (sensor_route_tables)

❗️

Caution

This section assumes familiarity with the ip command for managing route tables. Misconfiguring route tables could lock you out of your sensor host.

GreyNoise Sensors use custom route tables to route packets between the internet, the sensor, and the GreyNoise network.

The Sensor Agent can auto-repair these issues. If the issue persists, you can manually verify and repair the routes.

The sensor expects two custom routing tables (tables 3 and 4), each configured with one route:

• Table 3: default dev wg0 scope link
• Table 4: default via $GATEWAY dev $INTERFACE — where $GATEWAY is the default network gateway for the sensor's subnet, and $INTERFACE is the internet-facing network interface.

If these routes are missing, add them with ip route add. Remove any extra routes in these tables with ip route del. Do not flush or delete tables 3 and 4 — only add or remove individual routes within them as needed.

You can ignore other custom routing tables that may exist. If routing rules reference unexpected tables, that will be caught separately by the Route Rules check.

Wireguard (sensor_wireguard)

❗️

Caution

This section assumes familiarity with Wireguard and the ip command.

GreyNoise Sensors rely on Wireguard to securely route packets to the GreyNoise platform. If Wireguard isn't running or is misconfigured, packets will not reach GreyNoise.

Step 1: Verify the wg0 link exists and is up:

ip link show wg0

Step 2: If the link is missing or down, check that a config file exists at /etc/wireguard/wg0.conf. This file contains critical information generated during sensor initialization. If this file is missing or corrupted, the sensor will need to be recreated — see Quick Resolution above.

🚧

Key mismatch after reinstall

If you reinstalled the sensor software without first disabling the old sensor in the Visualizer, the reinstall generates a new Wireguard keypair that will not match the keys GreyNoise has on record. The Wireguard tunnel will silently fail to establish. To fix this, disable the sensor in the Visualizer, then reinstall from scratch.

Step 3: If the config file exists, check the wg-quick service:

systemctl status wg-quick@wg0

If it isn't running, enable and start it:

systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

If it is running, try restarting:

systemctl restart wg-quick@wg0

This should resolve the Wireguard issue. If the wg0 configuration is otherwise corrupted, this check may clear but the sensor may remain unhealthy for other reasons (typically Gateway Changed or GreyNoise Internal Error).

Gateway Changed (sensor_gateway_changed)

The gateway referenced here is the GreyNoise gateway that the sensor communicates with when routing packets to the GreyNoise platform. While Sensors rarely switch gateways, when they do, several components need to be updated on the Sensor side.

This status means the Sensor Agent has detected a gateway change but the sensor has not yet been reconfigured. The agent is responsible for performing this reconfiguration automatically, so the Sensor should return to a healthy state on its next check cycle.

If this check is persistently failing, verify the following:

1. The file /opt/greynoise/gateway.id exists and is world-readable.
2. The wg-quick@wg0 service is enabled and running.

If the issue continues to persist after verifying the above, the sensor is likely in an irrecoverable state and needs to be replaced — see Quick Resolution.

IP Changed (sensor_ip_changed)

This status means the sensor's public IP address has changed from what GreyNoise has on record. GreyNoise requires a stable, static IP address for each Sensor.

If your sensor's IP address changed, update it in the GreyNoise Visualizer on the sensor's detail page. See Sensor Management > Sensor Detail Page for details on editing the sensor IP.

You can also update the sensor IP via the API:

curl -X PUT \
  -H "key: $API_KEY" \
  -L "https://api.greynoise.io/v1/workspaces/$WORKSPACE_ID/sensors/$SENSOR_ID" \
  -d '{"public_ips": ["$NEW_IP"]}'

📘

Note

The public_ips field is an array. If your sensor has multiple public IPs, include all of them in the request.

🚧

Static IP Reminder

If your sensor is using a DHCP-provided IP address, we strongly recommend switching to a static IP. A changed IP will break the sensor's Wireguard tunnel and prevent it from communicating with the GreyNoise platform until the IP is updated.

SSH Port Mismatch (sensor_ssh_port_mismatch)

This status means the SSH management port configured on the sensor host does not match what GreyNoise has on record. This can happen if the SSH configuration was manually changed after installation. Reconfigure SSH on the host to use the port shown in the Visualizer or contact [email protected].

Invalid or Misconfigured Profile

This means there is an issue with the Profile assigned to the sensor.

If this is a Profile you provided to GreyNoise: You will need to debug the profile itself. A good starting point is ensuring that all of the open ports you specified when defining the Profile are actually open and listening when the image runs.

If this is a GreyNoise-provided Profile: Try remapping the sensor to a different profile in the Visualizer. Once the sensor becomes healthy with the new profile, you can try switching back to the original to determine if the issue was transient. If the sensor becomes unhealthy again with the original profile, contact [email protected] so we can investigate.

GreyNoise Internal Error

This means the issue is on the GreyNoise side rather than with your Sensor host. There are a few things you can try:

1. Remap the Sensor to a different profile. Allow a few minutes for the remap to complete (typically about 5 minutes) before checking the health status again. During this time, the Sensor may still appear unhealthy.
2. Replace the Sensor using any of the options in Quick Resolution.
3. Wait a few minutes. The issue may be transient, or our team may already be working on a fix. The sensor may recover on its own.

If the issue persists, contact [email protected] to report the problem.

Advanced: Packet Flow Debugging

If you've worked through the component-specific sections above and the sensor is still unhealthy, you can narrow down the problem by checking where packets are and aren't flowing. This requires familiarity with tcpdump.

Use tcpdump to capture traffic on both the main network interface and the Wireguard interface (wg0). Send a test request to your sensor's public IP on an open port, then check what you see:

• Packets on the main interface but NOT on wg0

  • The sensor is receiving traffic from the internet but not forwarding it through the Wireguard tunnel. This points to a routing or iptables issue on the sensor. Check the Route Rules, Route Tables, and IPTables sections.

• Packets on both interfaces outbound, but no response packets on wg0

  • Traffic is reaching GreyNoise but responses aren't coming back. This is a GreyNoise-side issue. Contact [email protected].

• Response packets on wg0 but NOT on the main interface

  • Responses are arriving from GreyNoise but aren't being forwarded back out to the internet. This is a return-path routing issue — check the Route Rules and Route Tables sections, paying particular attention to table 4 (the outbound route).

• Packets on both interfaces in both directions, but the sensor is still unhealthy

  • Packet flow is working correctly. The issue is likely with the profile or a GreyNoise-side configuration problem. See Invalid or Misconfigured Profile or GreyNoise Internal Error.

Example tcpdump commands to capture on each interface:

# Capture on the main interface (replace eth0 with your interface name)
sudo tcpdump -i eth0 host <SENSOR_PUBLIC_IP> -nn

# Capture on the Wireguard interface
sudo tcpdump -i wg0 -nn

General Troubleshooting FAQ

• Why am I unable to SSH into my sensor after installation?

  • The SSH port is changed during installation. The new port is provided in the installation script output. If you missed it, you can find it on the sensor detail page in the Visualizer. Connect with: ssh -p <your-ssh-port> user@<your-sensor-ip>

• Why does my sensor show the wrong IP address in the Visualizer?

  • On some hosts, the public IP address cannot be automatically detected during installation. Use the -i or -p parameter with the sensor installation script and redo the installation. See Customized Installation Using Optional Parameters.

• My sensor shows as "Connected" instead of "Healthy" — what does that mean?

  • "Connected" means the sensor's Wireguard tunnel is active but the Sensor Agent (greygent) is not installed. Without the agent, we can't collect detailed health data. See Missing Health Monitoring Agent to install it.

• My sensor was working but suddenly became unhealthy — what happened?

  • Common causes include: the sensor's public IP changed (especially with DHCP), another service or package was installed on the host that interfered with networking configuration, or a system update modified iptables rules or routing tables. Check the sensor detail page for the specific reason and follow the corresponding section in this guide.

• Can I run other software on my sensor host?

  • No. The sensor host must be dedicated to serving as a GreyNoise Sensor. Running other software — particularly anything that manages networking, firewalls, or routing — can interfere with sensor operation. This includes tools like NetworkManager, ufw, firewalld, or other firewall managers.

• How do I check if my firewall is blocking sensor traffic?

  • The firewall on your server must permit traffic on the ports that the assigned profile requires. We recommend allowing all inbound ports for maximum visibility. If you are running the sensor behind an external firewall or security group, ensure it also permits the required ports.

• I'm running my sensor on AWS EC2 and it's not working — what should I check?

  • Ensure source_dest_check is disabled on the EC2 instance's network interface (ENI). AWS enables this by default, and it will silently drop packets that don't match the instance's assigned IP — which breaks sensor operation. You can disable it in the EC2 console under the instance's networking settings, or via the CLI: aws ec2 modify-instance-attribute --instance-id $INSTANCE_ID --no-source-dest-check. Also ensure the instance has an Elastic IP (static) rather than a dynamic public IP, and that the security group allows all inbound traffic on the ports your profile requires.

Still Need Help?

For any issues not covered in this guide, or if you've worked through the relevant troubleshooting steps without resolution, contact [email protected] for assistance.