Observability
Overview
Whenever changes occur in your ngrok account or when traffic transits through your endpoints, an event is fired. You may subscribe to these events, filter them to those relevant to you and publish them to any number of destinations.
ngrok's event system was designed for three primary use cases:
- Sending logs of your ngrok traffic to logging services like Datadog and CloudWatch Logs
- Sending audit logs of ngrok configuration changes to your SIEM
- Enabling you to programmatically respond to events on your ngrok account
There are over 40 event types published via ngrok at the time of writing. They break down into two important classes of event:
- Traffic Events: When traffic transits through your endpoints like processing an HTTP request or TCP connection
- Audit Events: Changes to your account like create/update/delete of Domains, API Keys, IP Policies, etc.
Traffic Events are also captured in ngrok's Traffic Inspector, which provides an easy, built-in way to debug and test traffic flowing through your ngrok account.
Concepts
ngrok's event logging system is composed of three simple primitives. You create an Event Subscription to subscribe to a set of Event Sources and publish those events to one or more Event Destinations.
- Event Subscriptions: Define which Event Sources to capture and which Event Destinations to publish to.
- Event Sources: The ngrok event types to capture and optionally a set of fields to record and a boolean filter expression.
- Event Destinations: Where events are sent, e.g. Datadog, CloudWatch, Kinesis.
Quickstart
You can create your first Event Subscription on the Events page of your ngrok Dashboard to begin capturing and publishing events.
We also publish guides to get started with each of ngrok's Event Destinations:
Event Subscriptions
Event subscriptions define which Event Sources to capture and which destinations to publish to. If you're familiar with other event systems, they may call this a listener, a hook, a probe or a tap.
When you create an Event Subscription, you select:
- One or more Event Sources to subscribe to
- One or more Event Destinations to publish the subscribed events to
Event Subscriptions can be created on the ngrok Dashboard or via the Event Subscription API Resource.
Event Sources
Event Sources choose which events an Event Subscription captures. Other event systems may call this a selector.
The ngrok Event Sources reference documents the full list of Event Sources and their fields.
Versioning and Payload Stability
Event Sources include a version number in their name. Each Event Source ends
with a version string after a dot, e.g. api_key_created.v0
.
New fields may be added to the payloads of an Event Source without a version change. Any other change to the event representation will lead to the introduction of a new version of the Event Source.
Audit Events
ngrok emits Audit Events when changes are made to your account like create/update/delete of Domains, API Keys, IP Policies, etc.
Traffic Events
When traffic transits through your endpoints, ngrok emits Traffic Events like processing an HTTP request or TCP connection.
Traffic Events support the ability to select exactly which fields you'd like to include in the captured event. This provides you with control over how much data is sent to the destinations.
Traffic Events also support the ability to be filtered. As part of the Event Source definition you can specify a CEL expression which is evaluated against each event to determine if it should be captured. See the Filters section for further detail.
Filters
You may specify a filter on the Event Sources of Traffic Events. Filters are a boolean expression defined in Google's Common Expression Language (CEL). Filters are evaluated on each event as it is published to determine whether it should be sent to the Event Subscription's destinations.
Traffic Events are often high cardinality. We support filters because you may only interested in logging a subset of traffic (e.g. specific client IPs). Filters also enable you to control spend on downstream destinations where you incur costs based on the quantity of data processed and stored.
The event object is made available to the CEL filter expression as the ev
variable. ev
corresponds to the object
field of the captured event. Use
standard JSONPath syntax to access
the fields of the event you want to use in your filters.
Filter expressions have the following limitations:
- Only some fields of an event are available in the
ev
object. The Event Source's reference documentation will specify whether a field may be evaluated in a filter expression. - Event metadata fields like
principal
orevent_timestamp
are not available
Filter Examples
Filter for connections from a particular hostname
ev.conn.server_name.matches("ngrok-docs-examples\\.ngrok\\.dev")
Filter for connections to an endpoint that didn't use HTTPS
ev.conn.server_name.matches(".*-your-org\\.ngrok\\.dev") &&
ev.conn.server_port == 80
Filter for connections to a hostname that exclude traffic from a client IP
ev.conn.client_ip != "2601:0:8200:0:4cd7:fd52:0:7823" &&
ev.conn.server_name == "ngrok-docs-examples.ngrok.dev"
Event Destinations
An Event Destination encapsulates the configuration to publish events to other systems. Other event systems may call this a sink.
Event Destinations are typically third-party logging aggregators. The following destinations are currently supported:
- Amazon CloudWatch
- Amazon Firehose
- Amazon Kinesis
- AWS S3 (via Kinesis Firehose)
- Datadog Logs
Each destination requires provider-specific configuration. If you create a destination in the ngrok dashboard, you'll be prompted to send a test event to verify the integration.
When configuring AWS destinations you'll be prompted to optionally download a small helper script which will automatically configure the appropriate IAM objects necessary for integration. You may also set these values up via the AWS Console or tools like Terraform or Pulumi.
Amazon S3 is not a directly supported destination. Instead, configure Amazon Firehose to deliver events into an S3 bucket.
Event Payloads
Events are serialized as JSON when they are published to a destination.
Events include the following fields:
Name | Description | Example |
---|---|---|
account_id | unique identifier for the account, always prefixed with ac_ | ac_2OtNvAlhso10Gx6s7eupzX3F98q |
event_id | unique identifier for this event, always prefixed with ev_ | ev_1vPlyBW3OR44bpPphS4HIZyajDD |
event_type | identifies the object, action, and version of the event | ip_policy_created.v0 |
event_timestamp | timestamp of when the event fired in RFC 3339 format | 2021-07-16T21:44:37Z |
object | the event object | See examples below |
principal | an object of the principal who actioned this event, null for traffic events | See example below |
The object
property of the event is distinct for each Event Source.
For Audit Events, the object
representation is identical to its API resource at the time of capture.
Traffic Events define their own object
representation because they have no corresponding API resource definition.
Principal Object
The principal
object in every event describe the user or bot user responsible
for initiating the event. Principal is defined for all Audit Events
and it is null
for Traffic Events.
Name | Description | Example |
---|---|---|
id | unique identifier for the principal, either a user or bot | usr_2OtNv9qH5Nk4NuNeszZ39gBxZ4H |
subject | human readable unique identifier for the principal, either a user email or a bot user name | foo@example.com |
source | where the principal initiated the event, either 'Dashboard' or 'API' | API |
credential | id and uri resource for the credential used for authentication, null if source is 'Dashboard' | {"id": "ak_2Oxt94wYsBTLwFUoMZcJRvJTaub","uri": "https://api.ngrok.com/api_keys/ak_2Oxt94wYsBTLwFUoMZcJRvJTaub"} |
Example Payloads
ip_policy_created.v0 event
{
"event_id": "ev_25X2AsJ5xpvuOParTYUQWe12XKo",
"event_type": "ip_policy_created.v0",
"event_timestamp": "2022-02-23T23:29:29Z",
"account_id": "ac_2OtNvAlhso10Gx6s7eupzX3F98q",
"principal": {
"id": "usr_2OtNv9qH5Nk4NuNeszZ39gBxZ4H",
"subject": "foo@example.com",
"source": "API",
"credential": {
"id": "ak_2Oxt94wYsBTLwFUoMZcJRvJTaub",
"uri": "https://api.ngrok.com/api_keys/ak_2Oxt94wYsBTLwFUoMZcJRvJTaub"
}
},
"object": {
"id": "ipp_25X2Ao39z73FlVQKZ1iReMPe6Qv",
"uri": "https://api.ngrok.com/ip_policies/ipp_25X2Ao39z73FlVQKZ1iReMPe6Qv",
"created_at": "2022-02-23T23:29:29Z",
"description": "Home network IP",
"metadata": "",
"action": "allow"
}
}
http_request_complete.v0 event
{
"event_id": "ev_25X3yFS6TDkig1KDJWIc4nnJO0c",
"event_type": "http_request_complete.v0",
"event_timestamp": "2022-02-23T23:44:16Z",
"account_id": "ac_2OtNvAlhso10Gx6s7eupzX3F98q",
"object": {
"conn": {
"client_ip": "2601:0:8200:9e:4cd7:0:c97f:7823",
"server_name": "ngrok-docs-example.ngrok.app",
"server_port": ""
},
"http": {
"request": {
"first_byte_ts": null,
"last_byte_ts": null,
"method": "GET",
"url": {
"path": "/docs/obs"
},
"version": "HTTP/2.0"
},
"response": {
"body_length": 13079,
"first_byte_ts": "2022-02-23T23:44:16.732791273Z",
"last_byte_ts": "2022-02-23T23:44:16.737257209Z",
"status_code": 200
}
}
}
}
tcp_connection_closed.v0 event
{
"event_id": "ev_25X4osod1q306srserDeFyghTC4",
"event_type": "tcp_connection_closed.v0",
"event_timestamp": "2022-02-23T23:51:14Z",
"account_id": "ac_2OtNvAlhso10Gx6s7eupzX3F98q",
"object": {
"conn": {
"bytes_in": 3437,
"bytes_out": 90256,
"client_ip": "2601:0:8200:9e:4cd7:0:c97f:7823",
"end_ts": "2022-02-23T23:51:14.005372199Z",
"server_name": "ngrok-docs-example.ngrok.app",
"server_port": "",
"start_ts": "2022-02-23T23:44:16.528374173Z"
}
}
}
Pricing
Events are available to all ngrok users with a free tier that includes the transmission of up to 10,000 events per month.