Drill-down, naming, GeoIP upload, simpler Top-N

CHANGELOG.md

@@ -5,23 +5,48 @@ heading is the version number.
 
 ## 0.1.2
 
-Using it
-- New **Conversations** page in the web UI and the terminal UI (key 3): who
-  talks to whom, with client, server, service and traffic. The terminal UI's
-  pages are now 1–9.
-- The side menu no longer shows group headings that looked like buttons but
-  did nothing; groups are separated by lines.
+Drill-down and naming
+- **Show details** on any host, device or service opens a page about it:
+  traffic over time by application, who it talks to, services or clients,
+  countries and the latest flows. Everything on it can be clicked again to
+  keep drilling down; the browser's Back button returns.
+- **Name it…** on any host or device names it on the spot; the name is
+  saved and shown everywhere. The Names box on Sources now shows examples.
+
+Countries and networks database
+- Upload a database on the Sources page: MaxMind GeoLite2 or DB-IP Lite
+  `.mmdb` files (country or ASN) or an IP-to-ASN table (`.tsv`, `.tsv.gz`).
+  It is checked, saved and used for new traffic at once, without a
+  restart. The `.mmdb` reader was checked against MaxMind's own reader on
+  60,000 lookups in six DB-IP databases with no difference.
+
+Simpler pages
+- Top-N is one table: conversations (client, server, service, country) by
+  default; click a column heading to group by it, a number heading to
+  sort. The row of eleven tabs is gone; the less common groupings are in
+  **Group by**.
+- "Who grew" is gone from the overview (web and terminal UI).
+- Flow paths: labels, bars and flows can all be clicked; before, only the
+  thin bars could.
+- The side menu no longer shows group headings that looked like buttons;
+  "Log out" no longer wraps.
+- Disk in the side menu shows used and free space; hovering over the free
+  space shows how much the kept days of detail need at the current rate.
+  The old "days left" figure was an unreliable extrapolation.
+
+Capture
 - Windows local capture by name: `-capture Wi-Fi`, `-capture Ethernet` or
-  the number from `traffic66 interfaces`, instead of Npcap's
-  `\Device\NPF_{…}` names. `traffic66 interfaces` now shows each adapter's
+  the number from `traffic66 interfaces`, which now shows each adapter's
   Windows connection name and address.
+- Commands copied from a web page or chat work: full-width spaces, quotes,
+  full-width dashes and a trailing full stop around words are ignored, and
+  a mistyped command gets a "did you mean" hint.
 
 Documentation
-- README section "Local capture" rewritten with step-by-step instructions
-  for Windows, Linux and macOS, including what a Wi-Fi adapter can and
-  cannot see (13 languages).
-- This changelog; releases now carry these notes instead of a list of pull
-  requests.
+- README (13 languages): drill-down, naming, the database upload, local
+  capture step by step for Windows, Linux and macOS including what Wi-Fi
+  can and cannot see, new screenshots.
+- This changelog; releases take their notes from it.
 
 ## 0.1.1
 

README.md

@@ -18,7 +18,7 @@ in a web UI and in a terminal UI.
   flow records, encapsulation (GRE, IPIP, VXLAN, GENEVE, MPLS).
 - 13 languages in the web UI and the terminal UI.
 
-![Overview: bandwidth by application compared with last week, who grew, top clients and services](docs/images/overview.png)
+![Overview: bandwidth by application compared with last week, top clients and services](docs/images/overview.png)
 
 <sub>All screenshots come from `traffic66 demo`, a simulated company network that you can run yourself (see [Try the demo](#1-try-the-demo)).</sub>
 
@@ -84,7 +84,8 @@ cd traffic66-windows-amd64
 Open http://127.0.0.1:8066 and sign in as `admin` / `try66`. The demo builds
 a small company network with a day of history and live traffic from four
 simulated devices, including two incidents to find: start on **Overview**,
-look at **Who grew**, and click your way from there. Stop it with Ctrl+C.
+click a host in **Top clients**, choose **Show details**, and keep clicking
+from there. Stop it with Ctrl+C.
 Demo data is kept in `traffic66-demo` next to the program; delete that
 folder to start the demo afresh.
 
@@ -521,7 +522,12 @@ the sampling rate is not known yet.
 
 ## 7. Names, SNMP and your own networks
 
-**Sources → Names** in the web UI takes one entry per line. It is saved as
+The quickest way to name a host or a device: click its address on any page
+and choose **Name it…**. Type the name and press Enter; it is saved at
+once and shown everywhere instead of the bare address.
+
+For networks, interfaces and SNMP, **Sources → Names** in the web UI takes
+one entry per line. It is saved as
 `inventory.txt` in the data directory, so you can also edit that file (see
 `inventory.txt.example`). Every line is optional.
 
@@ -558,17 +564,27 @@ snmp   192.0.2.9     s3cret  10.99.0.9:161
 
 ## 8. Countries, networks and threat lists
 
-Countries and network (AS) names need an IP-to-ASN table. Download the free
-one from [iptoasn.com](https://iptoasn.com):
+Countries and network (AS) names need a database that maps addresses to
+them. Upload one in the web UI: **Sources → Countries and networks
+database → Upload a database file…**. It is checked, saved in the data
+directory and used for new traffic at once; no restart is needed. Traffic
+already stored keeps the country it was saved with.
 
-```
-curl -LO https://iptoasn.com/data/ip2asn-combined.tsv.gz
-mv ip2asn-combined.tsv.gz <data directory>/asn.tsv.gz
-```
+Accepted files:
 
-Any file in the same format works (tab separated: first address, last
-address, AS number, country code, AS name; plain or gzip). Restart
-traffic66 after replacing it; download a new one every month or so.
+| File | Gives | Where to get it |
+|---|---|---|
+| DB-IP Lite country or ASN, `.mmdb` | country, or AS number and name | free, no account: [db-ip.com/db/lite.php](https://db-ip.com/db/lite.php) |
+| MaxMind GeoLite2 Country or ASN, `.mmdb` | country, or AS number and name | free with a MaxMind account: [maxmind.com](https://www.maxmind.com/en/geolite2/signup) |
+| IP-to-ASN table, `.tsv` or `.tsv.gz` | AS number, AS name and country | free: [iptoasn.com](https://iptoasn.com) (`ip2asn-combined.tsv.gz`) |
+
+Upload a country database and an ASN database to get both; where several
+are loaded, the `.mmdb` files take precedence for what they contain. New
+versions come out monthly: upload the new file the same way to replace the
+old one.
+
+Without the web UI, copy the file into the data directory as
+`country.mmdb`, `asn.mmdb` or `asn.tsv.gz` and restart traffic66.
 
 Threat lists are plain text files with one address or network per line
 (text after `#` or `;` is ignored), saved as `<data directory>/threats/<name>.txt`, for
@@ -592,22 +608,28 @@ port, an application, a country, a device — can be clicked:
 - **Show only this** / **Exclude this** adds a filter. Filters appear
   under the top bar and apply to every page until you remove them.
 - **Show its flow records** opens the matching individual flows.
+- **Show details** (hosts, devices and services) opens a page about that
+  one host or service: its traffic over time by application, who it talks
+  to, which services or clients, countries and its latest flows. Every
+  value there can be clicked again, so you can keep drilling down; the
+  browser's Back button returns.
+- **Name it…** (hosts and devices) gives the address a name, shown
+  everywhere from then on.
 - **Look it up online** opens the address or AS in a public lookup site.
 - **Copy** copies the value.
 
 Pages:
 
 | Page | What it answers |
 |---|---|
-| Overview | How much traffic now and compared with last week, by application; what grew; top clients and services |
-| Top-N | The top 66 of clients, servers, conversations, applications, ports, countries, networks, segments, devices, encapsulation or VLAN |
-| Conversations | Who talks to whom: the top 66 client–server pairs with their service |
+| Overview | How much traffic now and compared with last week, by application; top clients and services |
+| Top-N | One table of the top 66: by default conversations (client, server, service, country). Click a blue column heading to group by it, a number heading to sort; **Group by** offers applications, networks, segments, devices, encapsulation and VLAN |
 | Flow paths | Which segment talks to which application in which country |
 | Geo & networks | Traffic by country and by network (AS) |
 | Threat intel | Hosts that talked to addresses on your threat lists, and how much they sent |
 | Flow records | Individual flows, newest first, with selectable columns |
 | Interface check | Flow numbers next to the interface counters, worst first, with reasons |
-| Sources | Devices, sampling, loss, collectors, SNMP, and **Names** |
+| Sources | Devices, sampling, loss, collectors, SNMP, the countries and networks database, and **Names** |
 
 Above the pages: time range (15 minutes to 30 days), an optional search
 box, automatic refresh every 30 seconds, and **Copy link**, which copies a
@@ -619,12 +641,16 @@ Ranges longer than 6 hours start on a whole hour, so every number on the
 page counts exactly the same time: "24 hours" covers the last 24 whole
 hours plus the current one. Top-N over these ranges comes from hourly
 summaries; filters are not available there, and the page says so. Choose a
-shorter range to filter. **Conversations** always reads the flow detail, so over long ranges
-at high flow rates it can take a while; one hour is fastest.
+shorter range to filter. Conversations always read the flow detail, so over long ranges
+at high flow rates they can take a while; one hour is fastest.
+
+The side menu shows how much disk the data uses and how much is free;
+hover over the free space to see how much the kept days of detail need at
+the current rate (estimated once there is a day of data).
 
-![Top-N: the top 66 clients of the last hour](docs/images/topn.png)
+![Top-N: the top 66 conversations of the last hour](docs/images/topn.png)
 
-![Conversations: who talks to whom, with the service](docs/images/conv.png)
+![Details of one host: its traffic, who it talks to, services, countries and latest flows](docs/images/detail.png)
 
 ![Flow paths: which segment uses which application towards which country](docs/images/paths.png)
 
@@ -646,13 +672,13 @@ traffic66 runs as another user, as a service does, use `-user` and
 `-password` instead. `-lang` picks the language (`en`, `zh`, `hi`,
 `es`, `ar`, `fr`, `bn`, `pt`, `ru`, `id`, `ur`, `ja`, `ko`).
 
-Keys: 1–9 pages, ↑↓ select, Enter actions on the selected value, f show
+Keys: 1–8 pages, ↑↓ select, Enter actions on the selected value, f show
 only, x exclude, / search, t time range, c clear filters, w open the same
 view in a browser, q quit.
 
 ![Terminal UI: overview](docs/images/tui-overview.png)
 
-![Terminal UI: Top-N clients](docs/images/tui-topn.png)
+![Terminal UI: Top-N conversations](docs/images/tui-topn.png)
 
 ## 11. Local capture
 
@@ -761,7 +787,7 @@ Options of `traffic66` and `traffic66 demo`:
 | `-sampling-wait` | `5m` | how long records wait for a sampling rate |
 | `-capture` | | capture on a local interface (repeatable) |
 | `-inventory` | `<data>/inventory.txt` | names file |
-| `-asn` | `<data>/asn.tsv.gz` | IP-to-ASN table |
+| `-asn` | `<data>/asn.tsv.gz` | IP-to-ASN table (`.mmdb` files: upload them, or `<data>/country.mmdb` and `<data>/asn.mmdb`) |
 | `-threat` | `<data>/threats/*.txt` | extra threat list as `name=path` (repeatable) |
 | `-dns-upstream` | system resolver | DNS server for showing host names |
 | `-dns-rate` | `20` | reverse lookups per second at most |
@@ -786,7 +812,7 @@ The data directory holds everything:
 | `traffic66.duckdb` | summaries, interface counters and the current hour |
 | `password` | login passwords (hashed) |
 | `inventory.txt` | names (**Sources → Names**) |
-| `asn.tsv.gz`, `threats/` | lookup tables you added |
+| `country.mmdb`, `asn.mmdb`, `asn.tsv.gz`, `threats/` | countries and networks databases and threat lists you added |
 
 - **Backup**: stop traffic66 and copy the directory. Without stopping,
   copy `raw/`, `password` and `inventory.txt`; the current hour and the
@@ -857,7 +883,7 @@ traffic66 then restarts instead of the machine running out of memory.
 | "waiting for the sampling rate" | The device has not sent its sampler options yet; most resend within minutes. If it never does, export them (`option sampler-table` on Cisco) or mark it `unsampled` in Names if it really is 1:1 |
 | Numbers lower than the interface counters | See **Interface check**: loss on the way, interfaces not sampled, or flows still in the device cache (active timeout longer than 60 s) |
 | Numbers higher than the interface counters | The same traffic sampled on two interfaces or two devices |
-| No countries or networks | No IP-to-ASN table: see [Countries](#8-countries-networks-and-threat-lists) |
+| No countries or networks ("Unknown") | No database loaded: upload one on **Sources**, see [Countries](#8-countries-networks-and-threat-lists) |
 | "The database reached its memory limit" on a page | Choose a shorter time range, or start with a larger `-memory`; details are in the log |
 | Forgot the password | `traffic66 passwd` on the traffic66 machine (add `-data` if traffic66 runs with it) |
 | `Conflicting lock is held` | Another traffic66 already uses this data directory |

cmd/traffic66/args_test.go

@@ -0,0 +1,20 @@
+package main
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestCleanArgs(t *testing.T) {
+	got := cleanArgs([]string{"　interfaces'.", "－capture", "“Wi-Fi”", " ", "-listen", ""})
+	want := []string{"interfaces", "-capture", "Wi-Fi", "-listen", ""}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("got %q want %q", got, want)
+	}
+	if s := nearestCommand("interface"); s != "interfaces" {
+		t.Fatalf("nearest %q", s)
+	}
+	if s := nearestCommand("zzzzzz"); s != "" {
+		t.Fatalf("nearest %q", s)
+	}
+}

cmd/traffic66/main.go

@@ -20,6 +20,8 @@ import (
 	"strings"
 	"syscall"
 	"time"
+	"unicode"
+	"unicode/utf8"
 
 	"github.com/githubflyideas/traffic66/internal/api"
 	"github.com/githubflyideas/traffic66/internal/auth"
@@ -28,6 +30,7 @@ import (
 	"github.com/githubflyideas/traffic66/internal/dnsres"
 	"github.com/githubflyideas/traffic66/internal/enrich"
 	"github.com/githubflyideas/traffic66/internal/flow"
+	"github.com/githubflyideas/traffic66/internal/geo"
 	"github.com/githubflyideas/traffic66/internal/pipeline"
 	"github.com/githubflyideas/traffic66/internal/sim"
 	"github.com/githubflyideas/traffic66/internal/snmp"
@@ -54,10 +57,10 @@ Run "traffic66 <command> -h" for the flags of a command.
 
 func main() {
 	log.SetFlags(log.LstdFlags)
-	args := os.Args[1:]
+	args := cleanArgs(os.Args[1:])
 	cmd := "serve"
 	if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
-		cmd, args = args[0], args[1:]
+		cmd, args = strings.ToLower(args[0]), args[1:]
 	}
 	switch cmd {
 	case "serve":
@@ -84,12 +87,79 @@ func main() {
 	case "help", "-h", "--help":
 		fmt.Printf(usage, version)
 	default:
-		fmt.Fprintf(os.Stderr, "unknown command %q\n\n", cmd)
+		fmt.Fprintf(os.Stderr, "unknown command %q", cmd)
+		if s := nearestCommand(cmd); s != "" {
+			fmt.Fprintf(os.Stderr, " (did you mean %q?)", s)
+		}
+		fmt.Fprint(os.Stderr, "\n\n")
 		fmt.Printf(usage, version)
 		os.Exit(2)
 	}
 }
 
+var commands = []string{"serve", "demo", "tui", "simulate", "passwd", "interfaces", "version", "help"}
+
+// cleanArgs undoes what copying a command from a web page or chat often adds:
+// full-width or non-breaking spaces, quotes around a word, full-width dashes
+// and a trailing full stop. Arguments that were only such characters are
+// dropped; an empty argument is kept.
+func cleanArgs(in []string) []string {
+	out := make([]string, 0, len(in))
+	for _, orig := range in {
+		if orig == "" { // an empty value given on purpose, e.g. -listen ""
+			out = append(out, orig)
+			continue
+		}
+		a := strings.TrimFunc(orig, func(r rune) bool {
+			return unicode.IsSpace(r) || strings.ContainsRune("'\"`‘’“”「」『』", r)
+		})
+		a = strings.TrimRight(a, ".。,，;；")
+		a = strings.TrimFunc(a, func(r rune) bool {
+			return unicode.IsSpace(r) || strings.ContainsRune("'\"`‘’“”「」『』", r)
+		})
+		if strings.HasPrefix(a, "－") || strings.HasPrefix(a, "—") || strings.HasPrefix(a, "–") {
+			_, n := utf8.DecodeRuneInString(a)
+			a = "-" + a[n:]
+		}
+		if a != "" {
+			out = append(out, a)
+		}
+	}
+	return out
+}
+
+// nearestCommand returns the command within two edits of s, if any.
+func nearestCommand(s string) string {
+	best, bestD := "", 3
+	for _, c := range commands {
+		if d := editDistance(s, c); d < bestD {
+			best, bestD = c, d
+		}
+	}
+	return best
+}
+
+func editDistance(a, b string) int {
+	ra, rb := []rune(a), []rune(b)
+	prev := make([]int, len(rb)+1)
+	for j := range prev {
+		prev[j] = j
+	}
+	for i := 1; i <= len(ra); i++ {
+		cur := make([]int, len(rb)+1)
+		cur[0] = i
+		for j := 1; j <= len(rb); j++ {
+			cost := 1
+			if ra[i-1] == rb[j-1] {
+				cost = 0
+			}
+			cur[j] = min(prev[j]+1, cur[j-1]+1, prev[j-1]+cost)
+		}
+		prev = cur
+	}
+	return prev[len(rb)]
+}
+
 type multiFlag []string
 
 func (m *multiFlag) String() string     { return strings.Join(*m, ",") }
@@ -175,8 +245,22 @@ func serve(args []string, demo bool) {
 		} else {
 			log.Printf("ASN table: %d ranges from %s", asn.Size(), asnPath)
 		}
-	} else {
-		log.Printf("no ASN/country table; put one at %s to see countries and networks", filepath.Join(f.data, "asn.tsv.gz"))
+	}
+	for _, name := range []string{"country.mmdb", "asn.mmdb"} {
+		p := filepath.Join(f.data, name)
+		if _, err := os.Stat(p); err != nil {
+			continue
+		}
+		r, err := geo.Open(p)
+		if err != nil {
+			log.Printf("%s: %v", p, err)
+			continue
+		}
+		asn.SetMMDB(r, name)
+		log.Printf("geo database %s: %s", name, r.Type)
+	}
+	if !asn.Loaded() {
+		log.Printf("no country/ASN database; upload one on the Sources page to see countries and networks")
 	}
 	thr := enrich.NewThreats()
 	threatFiles := map[string]string{}