traffic66 FILE.pcap: local offline analysis

CHANGELOG.md

@@ -16,6 +16,13 @@ Offline analysis
 - The detection rules run over the capture: scans, port scans and password
   guessing are found in it.
 - The demo includes an example capture with an attack.
+- `traffic66 capture.pcap` (up to 3 files, 3 GB in all) starts a private
+  traffic66 on 127.0.0.1 with a free port, prints the address, a password
+  and a one-time sign-in link, opens the browser on the capture, and
+  deletes the imported data on Ctrl+C. Files are read in place; nothing is
+  collected or sent and host names are not looked up unless `-dns` is
+  given. A 1 GB capture takes about 5 s (1.2 million full-size packets) to
+  30 s (14 million small packets) on 2 cores.
 
 Pages
 - Traffic details opens on servers only; tabs switch to clients, both ends

README.md

@@ -20,6 +20,7 @@ in a web UI and in a terminal UI.
 - Top 66 lists, traffic over time by client, server, service, interface
   and network (AS), flow paths, countries, threat list matches, flow
   records, encapsulation (GRE, IPIP, VXLAN, GENEVE, MPLS).
+- `traffic66 capture.pcap` opens up to 3 packet captures (3 GB in all) in the web UI: flows, findings, countries and flow records over the whole capture, with nothing to set up.
 - 13 languages in the web UI and the terminal UI.
 
 ![Overview: open findings, bandwidth by application compared with last week, top clients and services](docs/images/overview.png)
@@ -737,7 +738,30 @@ The same overview in Chinese; every page is available in 13 languages:
 
 ### Offline analysis
 
-**Offline analysis** looks at packet captures from Wireshark or tcpdump with the same pages as the live data, without mixing them in:
+**Offline analysis** looks at packet captures from Wireshark or tcpdump with the same pages as the live data, without mixing them in.
+
+It summarizes all the packets into flows: who talked to whom, how much, when, and what looks like an attack. It does not decode protocols or show packet contents; for one packet or one TCP stream, use Wireshark.
+
+From the command line, without setting anything up:
+
+```
+traffic66 office.pcap
+traffic66 a.pcap b.pcapng c.pcap
+```
+
+traffic66 starts on this computer only (127.0.0.1, a free port), prints the address, the password and a one-time sign-in link, and opens the browser on the capture. Up to 3 files, 3 GB in all; they are read where they are and never changed. Nothing is collected or sent, and host names are not looked up (`-dns` turns that on). Ctrl+C stops and deletes the imported data. On a 2-core machine a 1 GB capture is ready in about 5 seconds (1.2 million full-size packets) to 30 seconds (14 million small packets).
+
+```
+$ traffic66 office.pcap
+
+traffic66 0.3.1: analysing 1 capture file(s); nothing is collected or sent
+  Web UI    http://127.0.0.1:38217  (port 38217, this computer only)
+  Sign in   user admin, password gfhfhbuutz2e
+  Open      http://127.0.0.1:38217/auto?t=b9388f…  (signs in once)
+  Stop      Ctrl+C; the imported data is deleted, your files are kept
+```
+
+In the web UI of a running traffic66:
 
 1. **Upload capture files…**: `.pcap` or `.pcapng`, not compressed. Up to 3 files, each at most 50 MB. The files are turned into flows in a database of their own (`<data>/sandbox/`); the live data, its numbers and findings are not touched.
 2. **Analyse**: every page (overview, Top 66, traffic details, findings, flow paths, map, flow records) now shows the capture files over their whole time. An orange bar names the files; **Back to live data** returns. Each file appears as a device, so the **Device** box shows one file at a time.

cmd/traffic66/main.go

@@ -18,6 +18,7 @@ import (
 	"os/signal"
 	"path/filepath"
 	"runtime/debug"
+	"slices"
 	"strings"
 	"syscall"
 	"time"
@@ -47,6 +48,7 @@ const usage = `traffic66 %s — flow analytics for sFlow, NetFlow and IPFIX
 
 Usage:
   traffic66 [serve] [flags]     collect flows and serve the web UI (default)
+  traffic66 FILE.pcap [...]     analyse up to 3 capture files (pcap, pcapng) in the web UI
   traffic66 demo [flags]        run with a built-in simulated network
   traffic66 tui [flags]         terminal UI (connects to a running traffic66)
   traffic66 simulate -to HOST   send simulated exports to another collector
@@ -60,6 +62,10 @@ Run "traffic66 <command> -h" for the flags of a command.
 func main() {
 	log.SetFlags(log.LstdFlags)
 	args := cleanArgs(os.Args[1:])
+	if len(args) > 0 && !strings.HasPrefix(args[0], "-") && !slices.Contains(commands, strings.ToLower(args[0])) && looksLikeCapture(args[0]) {
+		runOffline(args)
+		return
+	}
 	cmd := "serve"
 	if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
 		cmd, args = strings.ToLower(args[0]), args[1:]
@@ -389,7 +395,18 @@ func serve(args []string, demo bool) {
 	srv := &api.Server{Store: st, Pipe: pipe, Col: col, Inv: inv, ASN: asn, Thr: thr, DNS: dns, Det: det, Static: web.FS(), Version: version,
 		Demo: demo, Check: checker.Check, Exists: checker.Exists, LocalTok: tok, DataDir: f.data, Started: time.Now()}
 	srv.SNMP = poller.Status
-	srv.SB = sandbox.New(filepath.Join(f.data, "sandbox"), inv, asn, thr)
+	if offline != nil {
+		srv.SB = sandbox.NewWith(filepath.Join(f.data, "sandbox"), inv, asn, thr, sandbox.LocalLimits, 0.25)
+		srv.Offline = true
+		srv.AutoLogin = randomHex(16)
+		for _, p := range offline.files {
+			if _, err := srv.SB.AddPath(p); err != nil {
+				fatalf("%v", err)
+			}
+		}
+	} else {
+		srv.SB = sandbox.New(filepath.Join(f.data, "sandbox"), inv, asn, thr)
+	}
 	defer srv.SB.Close()
 	if demo {
 		demoSample(f.data, srv.SB)
@@ -408,6 +425,18 @@ func serve(args []string, demo bool) {
 	hs := &http.Server{Handler: srv.Handler(), ReadHeaderTimeout: 10 * time.Second}
 	go hs.Serve(ln)
 	log.Printf("web UI: http://%s", displayAddr(ln.Addr()))
+	if offline != nil {
+		u := "http://" + displayAddr(ln.Addr())
+		_, port, _ := net.SplitHostPort(ln.Addr().String())
+		fmt.Printf("\ntraffic66 %s: analysing %d capture file(s); nothing is collected or sent\n", version, len(offline.files))
+		fmt.Printf("  Web UI    %s  (port %s, this computer only)\n", u, port)
+		fmt.Printf("  Sign in   user %s, password %s\n", f.user, f.password)
+		fmt.Printf("  Open      %s/auto?t=%s  (signs in once)\n", u, srv.AutoLogin)
+		fmt.Printf("  Stop      Ctrl+C; the imported data is deleted, your files are kept\n\n")
+		if offline.browser {
+			tui.OpenBrowser(u + "/auto?t=" + srv.AutoLogin)
+		}
+	}
 	if doubleClick {
 		log.Printf("opening the web UI in your browser; close this window to stop traffic66")
 		tui.OpenBrowser("http://" + displayAddr(ln.Addr()))

cmd/traffic66/offline.go

@@ -0,0 +1,124 @@
+package main
+
+import (
+	"crypto/rand"
+	"flag"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/githubflyideas/traffic66/internal/pcapfile"
+	"github.com/githubflyideas/traffic66/internal/sandbox"
+)
+
+// offlineRun holds what "traffic66 file.pcap" passes to serve.
+type offlineRun struct {
+	files   []string
+	browser bool
+}
+
+var offline *offlineRun
+
+// looksLikeCapture tells "traffic66 file.pcap" from a mistyped command.
+func looksLikeCapture(arg string) bool {
+	ext := strings.ToLower(filepath.Ext(arg))
+	if ext == ".pcap" || ext == ".pcapng" || ext == ".cap" {
+		return true
+	}
+	fi, err := os.Stat(arg)
+	return err == nil && fi.Mode().IsRegular()
+}
+
+// checkCaptures checks the files before anything starts: at most the local
+// limits, and each a pcap or pcapng file.
+func checkCaptures(files []string, lim sandbox.Limits) error {
+	if len(files) > lim.Files {
+		return fmt.Errorf("at most %d files at a time (got %d)", lim.Files, len(files))
+	}
+	var total int64
+	for _, p := range files {
+		fi, err := os.Stat(p)
+		if err != nil {
+			return err
+		}
+		if !fi.Mode().IsRegular() {
+			return fmt.Errorf("%s is not a file", p)
+		}
+		f, err := os.Open(p)
+		if err != nil {
+			return err
+		}
+		head := make([]byte, 4)
+		_, err = io.ReadFull(f, head)
+		f.Close()
+		if err != nil || pcapfile.Format(head) == "" {
+			return fmt.Errorf("%s is not a capture file: use .pcap or .pcapng as saved by Wireshark or tcpdump (not compressed)", p)
+		}
+		total += fi.Size()
+	}
+	if total > lim.TotalSize {
+		return fmt.Errorf("the files have %s; at most %s in all", humanBytes(total), humanBytes(lim.TotalSize))
+	}
+	return nil
+}
+
+func humanBytes(n int64) string {
+	switch {
+	case n >= 1e9:
+		return fmt.Sprintf("%.1f GB", float64(n)/1e9)
+	case n >= 1e6:
+		return fmt.Sprintf("%.1f MB", float64(n)/1e6)
+	}
+	return fmt.Sprintf("%d bytes", n)
+}
+
+// runOffline analyses capture files: a private traffic66 on 127.0.0.1 with
+// a temporary database that is deleted on exit; the files stay untouched.
+func runOffline(args []string) {
+	var files, rest []string
+	for i, a := range args {
+		if strings.HasPrefix(a, "-") {
+			rest = args[i:]
+			break
+		}
+		files = append(files, a)
+	}
+	fs := flag.NewFlagSet("traffic66 FILE.pcap", flag.ExitOnError)
+	addr := fs.String("addr", "127.0.0.1:0", "web UI address (default: a free port on this computer only)")
+	noBrowser := fs.Bool("no-browser", false, "do not open the browser")
+	dns := fs.Bool("dns", false, "look up host names of the addresses (off: a capture's addresses are not sent to DNS)")
+	fs.Usage = func() {
+		fmt.Fprintf(os.Stderr, "Usage: traffic66 FILE.pcap [FILE2.pcapng FILE3.pcap] [flags]\n\nAnalyse capture files (at most %d, %s in all) in the web UI.\n\n", sandbox.LocalLimits.Files, humanBytes(sandbox.LocalLimits.TotalSize))
+		fs.PrintDefaults()
+	}
+	fs.Parse(rest)
+	files = append(files, fs.Args()...)
+	if err := checkCaptures(files, sandbox.LocalLimits); err != nil {
+		fmt.Fprintln(os.Stderr, "traffic66:", err)
+		os.Exit(2)
+	}
+	tmp, err := os.MkdirTemp("", "traffic66-offline-")
+	if err != nil {
+		fatalf("temporary directory: %v", err)
+	}
+	defer os.RemoveAll(tmp)
+	offline = &offlineRun{files: files, browser: !*noBrowser}
+	sargs := []string{"-data", tmp, "-listen", "", "-addr", *addr, "-password", readablePassword(), "-memory", "0.15"}
+	if !*dns {
+		sargs = append(sargs, "-no-dns")
+	}
+	serve(sargs, false)
+}
+
+// readablePassword is 12 letters and digits without look-alikes.
+func readablePassword() string {
+	const set = "abcdefghjkmnpqrstuvwxyz23456789"
+	b := make([]byte, 12)
+	rand.Read(b)
+	for i := range b {
+		b[i] = set[int(b[i])%len(set)]
+	}
+	return string(b)
+}

cmd/traffic66/offline_test.go

@@ -0,0 +1,46 @@
+package main
+
+import (
+	"errors"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/githubflyideas/traffic66/internal/sandbox"
+)
+
+func TestCheckCaptures(t *testing.T) {
+	dir := t.TempDir()
+	pcap := []byte{0xd4, 0xc3, 0xb2, 0xa1, 2, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff, 0, 0, 1, 0, 0, 0}
+	write := func(name string, b []byte) string {
+		p := filepath.Join(dir, name)
+		os.WriteFile(p, b, 0o644)
+		return p
+	}
+	a, b, c, d := write("a.pcap", pcap), write("b.pcapng", append([]byte{0x0a, 0x0d, 0x0d, 0x0a}, make([]byte, 100)...)), write("c.cap", pcap), write("d.pcap", pcap)
+	txt := write("notes.txt", []byte("hello world"))
+	lim := sandbox.Limits{Files: 3, FileSize: 1000, TotalSize: 140}
+	for _, tc := range []struct {
+		files []string
+		err   string
+	}{
+		{[]string{a, b}, ""},
+		{[]string{a, b, c, d}, "at most 3 files"},
+		{[]string{txt}, "not a capture file"},
+		{[]string{a, b, c}, "in all"},
+	} {
+		err := checkCaptures(tc.files, lim)
+		if (tc.err == "") != (err == nil) || err != nil && !strings.Contains(err.Error(), tc.err) {
+			t.Errorf("%v: %v, want %q", tc.files, err, tc.err)
+		}
+	}
+	// the message differs between systems
+	if err := checkCaptures([]string{filepath.Join(dir, "missing.pcap")}, lim); !errors.Is(err, fs.ErrNotExist) {
+		t.Errorf("missing file: %v", err)
+	}
+	if !looksLikeCapture("x.PCAPNG") || looksLikeCapture("serv") || !looksLikeCapture(txt) {
+		t.Error("looksLikeCapture")
+	}
+}

docs/README.ar.md

@@ -22,6 +22,7 @@
 - قوائم أعلى 66، والحركة عبر الزمن حسب العميل والخادم والخدمة والواجهة
   والشبكة (AS)، ومسارات الحركة، والدول، والتطابقات مع قوائم التهديدات،
   وسجلات التدفق، والتغليف (GRE وIPIP وVXLAN وGENEVE وMPLS).
+- يفتح `traffic66 capture.pcap` حتى 3 ملفات التقاط (3 GB إجمالاً) في واجهة الويب: التدفقات والاكتشافات والدول وسجلات التدفق لكامل الالتقاط، دون أي إعداد.
 - 13 لغة في واجهة الويب والواجهة الطرفية.
 
 ![نظرة عامة: الاكتشافات المفتوحة، واستهلاك عرض النطاق حسب التطبيق مقارنةً بالأسبوع الماضي، وأبرز العملاء والخدمات](images/overview.png)
@@ -750,7 +751,30 @@ curl -L https://www.spamhaus.org/drop/drop.txt -o <data directory>/threats/spamh
 
 ### التحليل دون اتصال
 
-يعرض **التحليل دون اتصال** ملفات التقاط Wireshark أو tcpdump بالصفحات نفسها المستخدمة للبيانات الحية، دون خلطها بها:
+يعرض **التحليل دون اتصال** ملفات التقاط Wireshark أو tcpdump بالصفحات نفسها المستخدمة للبيانات الحية، دون خلطها بها.
+
+يلخّص كل الحزم في تدفقات: من تحدث مع من، وكم، ومتى، وما الذي يبدو هجوماً. لا يفك ترميز البروتوكولات ولا يعرض محتوى الحزم؛ لحزمة واحدة أو تدفق TCP واحد استخدم Wireshark.
+
+من سطر الأوامر، دون أي إعداد:
+
+```
+traffic66 office.pcap
+traffic66 a.pcap b.pcapng c.pcap
+```
+
+يبدأ traffic66 على هذا الحاسوب فقط (127.0.0.1، منفذ متاح)، ويطبع العنوان وكلمة المرور ورابط دخول لمرة واحدة، ويفتح الالتقاط في المتصفح. حتى 3 ملفات، 3 GB إجمالاً؛ تُقرأ الملفات في مكانها ولا تُعدَّل أبداً. لا يُجمع شيء ولا يُرسل، ولا يُبحث عن أسماء المضيفين (يفعّلها `-dns`). يوقف Ctrl+C البرنامج ويحذف البيانات المستوردة. على جهاز ثنائي النواة يجهز التقاط بحجم 1 GB في نحو 5 ثوانٍ (1.2 مليون حزمة كاملة الحجم) إلى 30 ثانية (14 مليون حزمة صغيرة).
+
+```
+$ traffic66 office.pcap
+
+traffic66 0.3.1: analysing 1 capture file(s); nothing is collected or sent
+  Web UI    http://127.0.0.1:38217  (port 38217, this computer only)
+  Sign in   user admin, password gfhfhbuutz2e
+  Open      http://127.0.0.1:38217/auto?t=b9388f…  (signs in once)
+  Stop      Ctrl+C; the imported data is deleted, your files are kept
+```
+
+في واجهة الويب لـ traffic66 قيد التشغيل:
 
 1. **رفع ملفات التقاط…**: ‎`.pcap` أو ‎`.pcapng` غير مضغوطة. حتى 3 ملفات، كل منها 50 MB كحد أقصى. تتحول الملفات إلى تدفقات في قاعدة بيانات خاصة بها (`<data>/sandbox/`)، فلا تتأثر البيانات الحية وأرقامها واكتشافاتها.
 2. **تحليل**: تعرض كل الصفحات (نظرة عامة، أعلى 66، تفاصيل الحركة، الاكتشافات، مسارات الحركة، الخريطة، سجلات التدفق) الملفات على امتداد وقتها كله. يذكر شريط برتقالي أسماء الملفات، و**العودة إلى البيانات الحية** يعيدك. يظهر كل ملف كجهاز، فيعرض مربع **الجهاز** ملفاً واحداً في كل مرة.