Exported data¶
Storing exported data¶
DNS Probe supports storing the exported data either to local files or transferring them directly to a remote location via secure network transfer using TLS. This is determined by the location option in YAML configuration file.
Local storage¶
If location option is set to local the exported data will be stored in local files in directory
specified by export-dir option. The names of these files will have the following naming convention:
<prefix>yyyyMMdd.HHmmss.SSSSSS.p<proc_id>.<sufix>
The <prefix> is determined by file-name-prefix option in YAML configuration file. The
yyyyMMdd.HHmmss.SSSSSS represents a UTC timestamp (microsecond precision) from when the output file was
first opened. <proc_id> is an internal identification of process (worker or export thread) which wrote
the output file. <sufix> is one of parquet, cdns or cdns.gz based on the export format and
compression configured in YAML file.
Export to remote location¶
If location option is set to remote DNS Probe will attempt to transfer the exported data to a remote
server specified by remote-ip-address and remote-port options via encrypted TLS connection with
remote server’s authentication. In case of connection issues with remote server there’s also a possibility to
configure backup remote server with options backup-remote-ip-address and backup-remote-port.
DNS Probe will attempt to send exported data to this backup remote server only if connection to primary remote
server fails.
The transfer uses a simple application layer protocol shown below:
0 1 2 3 4 5 6 7
+--+--+--+--+--+--+--+--+
| File name's length |
+--+--+--+--+--+--+--+--+
| File's name |
| ... |
+--+--+--+--+--+--+--+--+
| File's data |
| ... |
| ... |
+--+--+--+--+--+--+--+--+
Each file is transferred using a new TLS connection. The first byte of data determines the length of transferred file’s name. Then the name of the transferred file follows. The transferred file’s name follows the same convention as files saved by DNS Probe locally meaning that the remote server can determine the file’s timestamp, format and compression just from parsing the file’s name. After the file’s name the exported DNS data in Parquet or C-DNS format follows until the end of the connection. By correctly closing the connection DNS Probe signals to remote server that all the data has been sent and remote server can finish the currently transferred file.
To prevent a loss of data due to network outages DNS Probe stores data for the current file to a local directory, specified in export-dir, first. When the file is finished and DNS Probe is about to perform output rotation, then the probe tries to transfer the finished file to remote server. If the transfer succeeds the local file is deleted. DNS Probe will initially attempt to transfer the file three times. If all three transfer attempts fail the local file is kept. DNS Probe keeps track of files that failed the transfer to remote server and periodically tries to resend them. The local files are kept until such transfer is successful.
Data schema¶
DNS Probe exports data in one of two formats - Parquet or C-DNS. The exported data tries to conform to the Entrada schema for Hadoop. Parquet export simply copies the Entrada schema shown in the table below. C-DNS format has its own schema defined in RFC 8616. DNS Probe tries to fill this C-DNS schema with only the data needed for reconstructing the Entrada schema. As of DNS Probe release 1.3.0 export to C-DNS also has an option to export full resource records from Answer and Additional sections of responses by enabling cdns-export-response-rr configuration option.
| Entrada (Parquet) fields | type | C-DNS schema field | Comment |
|---|---|---|---|
| id | INT32 | transaction-id | 16-bit DNS ID |
| unixtime | INT64 | earliest-time, time-offset | Seconds since Epoch |
| time | INT64 | earliest-time, time-offset | Microseconds since Epoch |
| qname | STRING | query-name-index | Full qname, without final dot |
| domainname | STRING | query-name-index | Last two domains (or TLD + 1 label) in lowercase |
| len | INT32 | XXX | Request packet length |
| frag | INT32 | XXX | Fragmentation? (Always 0 in Parquet) |
| ttl | INT32 | client-hoplimit | Request TTL |
| ipv | INT32 | qr-transport-flags | IP version: 4 / 6 |
| prot | INT32 | qr-transport-flags | TCP/UDP/… (value such as “17” =UDP) |
| src | STRING | client-address-index | Source (client) IP |
| srcp | INT32 | client-port | Source (client) port |
| dst | STRING | server-address-index | Destination (server) IP |
| dstp | INT32 | server-port | Destination (server) port |
| udp_sum | INT32 | XXX | UDP checksum |
| dns_len | INT32 | query-size | Request DNS payload length |
| aa | BOOLEAN | qr-dns-flags | Response AA flag |
| tc | BOOLEAN | qr-dns-flags | Response TC flag |
| rd | BOOLEAN | qr-dns-flags | Request RD flag |
| ra | BOOLEAN | qr-dns-flags | Request RA flag |
| z | BOOLEAN | qr-dns-flags | Request Z flag |
| ad | BOOLEAN | qr-dns-flags | Request AD flag |
| cd | BOOLEAN | qr-dns-flags | Request CD flag |
| ancount | INT32 | query-ancount | Answers count |
| arcount | INT32 | query-arcount | Additional records count |
| nscount | INT32 | query-nscount | Authority records count |
| qdcount | INT32 | query-qdcount | Questions count |
| opcode | INT32 | query-opcode | Request opcode (=response) |
| rcode | INT32 | response-rcode | Response code |
| qtype | INT32 | type | Query type |
| qclass | INT32 | class | Query class |
| country | STRING | country-code (implementation field) | 2 letter code (“CZ”, ..) |
| asn | STRING | asn (implementation field) | ASN (“AS1234”, …) |
| edns_udp | INT32 | query-udp-size | UDP payload |
| edns_version | INT32 | query-edns-version | EDNS version |
| edns_do | BOOLEAN | qr-dns-flags | DO bit |
| edns_ping | BOOLEAN | response-extended.additional-index | Tough to detect! (always false in Parquet) |
| edns_nsid | STRING | response-extended.additional-index | NSID string |
| edns_dnssec_dau | STRING | query-opt-rdata-index | Comma-separated list “1,3,5” |
| edns_dnssec_dhu | STRING | query-opt-rdata-index | Comma-separated list “1,3,5” |
| edns_dnssec_n3u | STRING | query-opt-rdata-index | Comma-separated list “1,3,5” |
| edns_client_subnet | STRING | query-opt-rdata-index | Always empty string in Parquet |
| edns_other | STRING | query-opt-rdata-index | Always empty string in Parquet |
| edns_client_subnet_asn | STRING | query-opt-rdata-index | By IP list (Maxmind) (always empty string in Parquet) |
| edns_client_subnet_country | STRING | query-opt-rdata-index | By IP list (Maxmind) (always empty string in Parquet) |
| labels | INT32 | query-name-index | Number of qname labels |
| res_len | INT32 | XXX | Response packet length |
| time_micro | INT64 | earliest-time, time-offset | Microseconds part of time field |
| resp_frag | INT32 | XXX | Unknown (always 0 in Parquet) |
| proc_time | INT32 | XXX | Unknown (always 0 in Parquet) |
| is_google | BOOLEAN | XXX | By IP list (Maxmind) (always false in Parquet) |
| is_opendns | BOOLEAN | XXX | By IP list (Maxmind) (always false in Parquet) |
| dns_res_len | INT32 | response-size | Response DNS payload length |
| server_location | STRING | XXX | Server location (allways empty string in Parquet) |
| tcp_hs_rtt | DOUBLE | round-trip-time (implementation field) (microsecond precision integer) | TCP Round Trip Time (RTT) (milliseconds with up to 3 decimal digits precision) |
Storing exported run-time statistics¶
DNS Probe supports periodically storing run-time statistics in JSON format to local files or transferring them directly to a remote location via secure network transfer using TLS. This is determined by the location option in YAML configuration file.
Local storage¶
If location option is set to local the exported statistics will be stored in local files
in directory specified by export-dir option. The names of these files will have the following
naming convention:
<prefix>yyyyMMdd.HHmmss.SSSSSS.stats.json
The <prefix> is determined by file-name-prefix option in YAML configuration file. The yyyyMMdd.HHmmss.SSSSSS represents a UTC timestamp (microsecond precision) from when the JSON was exported.
Export to remote location¶
If location is set to remote DNS Probe will attempt to transfer the exported statistics
to a remote server specified by remote-ip and remote-port options
via encrypted TLS connection with remote server’s authentication. In case of connection issues with remote
server there’s also a possibility to configure backup remote server with options backup-remote-ip
and backup-remote-port. DNS Probe will attempt to send exported statistics to this backup remote
server only if connection to primary remote server fails.
The transfer uses the same simple application protocol used for traffic data transfer that was shown in chapter Export to remote location.
Exported statistics¶
DNS Probe can export the following run-time statistics:
| Statistics fields | Type | Comment |
|---|---|---|
| processed-packets | UINT64 | Cumulative number of processed packets since the start of probe |
| processed-transactions | UINT64 | Cumulative number of processed transactions (paired request-response) since the start of probe |
| exported-records | UINT64 | Cumulative number of exported DNS records (C-DNS/Parquet) since the start of probe |
| pending-transactions | UINT64 | Number of transactions currently waiting to be matched in transaction table |
| exported-pcap-packets | UINT64 | Cumulative number of packets exported to PCAP since the start of probe |
| ipv4-source-entropy | DOUBLE | Entropy for highest byte of client IPv4 addresses for the last stats-timeout seconds |
| queries-ipv4 | UINT64 | Cumulative number of IPv4 DNS requests seen on wire since the start of probe |
| queries-ipv6 | UINT64 | Cumulative number of IPv6 DNS requests seen on wire since the start of probe |
| queries-tcp | UINT64 | Cumulative number of TCP/53 DNS requests seen on wire since the start of probe |
| queries-udp | UINT64 | Cumulative number of UDP DNS requests seen on wire since the start of probe |
| queries-dot | UINT64 | Cumulative number of DoT DNS requests seen on wire since the start of probe |
| queries-doh | UINT64 | Cumulative number of DoH DNS requests seen on wire since the start of probe |
| queries | UINT64 | Cumulative number of DNS requests seen on wire since the start of probe |
| queries-per-second-ipv4 | UINT64 | Moving average (default 5 minutes) of IPv4 requests per second for the last moving-avg-window seconds |
| queries-per-second-ipv6 | UINT64 | Moving average (default 5 minutes) of IPv6 requests per second for the last moving-avg-window seconds |
| queries-per-second-tcp | UINT64 | Moving average (default 5 minutes) of TCP/53 requests per second for the last moving-avg-window seconds |
| queries-per-second-udp | UINT64 | Moving average (default 5 minutes) of UDP requests per second for the last moving-avg-window seconds |
| queries-per-second-dot | UINT64 | Moving average (default 5 minutes) of DoT requests per second for the last moving-avg-window seconds |
| queries-per-second-doh | UINT64 | Moving average (default 5 minutes) of DoH requests per second for the last moving-avg-window seconds |
| queries-per-second | UINT64 | Moving average (default 5 minutes) of requests per second for the last moving-avg-window seconds |
| unix-timestamp | UINT64 | Unix timestamp in microseconds at the moment this JSON was exported |
The table above represents run-time statistics exported with export-stats option set to true and stats-per-ip set to false.
If stats-per-ip option is set to true, all above statistics are exported and in addition if any
IP addresses are set in ipv4-allowlist or ipv6-allowlist, all queries* statistics are also
exported for each of the IP addresses in format "[<IP-address>]queries*":<value>.
