diff --git a/docs/plugins.md b/docs/plugins.md index 2741a7c2..2cf988c1 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -227,6 +227,7 @@ Unknown keys are ignored, so a manifest written for a later Folio still loads. | `view` | A view of all the plugin's items in the main pane (see [The view](#the-view)). | | `actions` | Entries in the right-click menu, in the menu itself or under Share (see [Actions](#actions)). | | `details` | Rows in the details pane about a file or folder (see [Details](#details)). | +| `watch` | `{"command": [...]}`: a command that stays running and says when something changed, so Folio asks again at once (see [Watching](#watching)). | ## Running commands @@ -265,6 +266,7 @@ How long a command may take depends on what it's for: waits until the first is done. Closing Folio ends it. - An action with `"detach": true` is started and forgotten, for programs that open a window of their own. +- A **watch** runs for as long as Folio does (see [Watching](#watching)). ### Text @@ -341,6 +343,43 @@ row, the one whose id sorts first is shown. Leave it out and every action is available. A plugin with a poll offers no actions while its poll is failing. +## Watching + +```json +"watch": { "command": ["./watch"] } +``` + +A poll asks on a timer, and a file's details are asked for once and kept, so +a plugin whose state changes on its own (a sync finishing, a machine coming +online) shows it late, or not until F5. A watch says when: Folio starts the +command when the plugins are read and leaves it running, and every line it +prints means something changed. Folio then runs the plugin's poll at once, +whatever its `every`, and asks its details command again about the file the +details pane shows. + +```sh +#!/bin/sh +# Says so whenever something in the notes folder changes. +exec inotifywait -m -q -r -e close_write,move,create,delete --format changed "$HOME/Notes" +``` + +What a line says isn't read, only that it came, so print whatever helps when +you run the command yourself. The first line is acted on at once; lines that +follow within half a second are acted on together when it's over, so a +command may print one for every little change without Folio polling for +each. A change that comes while a poll is running has the poll run once more +after it. + +The command is started like any other (in the plugin's folder, with +`FOLIO_PLUGIN_DIR` and `FOLIO_LANGUAGE`, and its standard error going to +Folio's), but with no time limit. It ends, with everything it started, when +Folio closes or the plugin goes away; F5 and a change of language leave a +running one be, unless its manifest's `watch` changed. When it ends by +itself (what it watches isn't running), Folio starts it again 5 seconds +later, and again every 5 seconds until it stays, saying so on its standard +error once. A plugin works without its watch, only later: keep the poll, and +let the watch make it timely. + ## The sidebar section ```json @@ -474,9 +513,9 @@ is fine when there's nothing to say. The rows show below Folio's own, plugins in id order. Folio keeps them for the file as it was last modified, so going through a folder and back runs nothing again. It asks again about a file changed since, about everything -after F5, and about every file once the plugin's poll prints something new, -since a plugin's state can change how a file stands without changing the -file (Dropbox finishing a sync). +after F5, and about every file once the plugin's poll prints something new +or its [watch](#watching) prints a line, since a plugin's state can change +how a file stands without changing the file (Dropbox finishing a sync). The command runs in the background, never holding up the window, and has 2 seconds. One that fails, takes longer or prints something other than a list @@ -589,7 +628,8 @@ file in `~/Nextcloud` stands. { "id": "share", "label": "Nextcloud…", "menu": "share", "command": ["./share"], "under": ["~/Nextcloud/"], "single": true } ], - "details": { "command": ["./filestatus"], "under": ["~/Nextcloud/"] } + "details": { "command": ["./filestatus"], "under": ["~/Nextcloud/"] }, + "watch": { "command": ["./watch"] } } ``` @@ -603,7 +643,10 @@ it syncs (no status: the client shows its own) and prints `"actions": `share` sends the client `SHARE`, for it to open its share options in a window of its own. `filestatus` prints `[["Nextcloud", status]]` with `up to date`, `syncing`, `ignored` or `error`, then `, shared` for what's shared, and `[]` -when the client isn't running or doesn't sync the file. A manifest's `under` +when the client isn't running or doesn't sync the file. `watch` stays on the +socket with `socat` and passes on what the client says, a line whenever +something in a folder it syncs changed, so the details row keeps up with a +sync as it goes; with no client running it ends, and Folio starts it again. A manifest's `under` can't follow the client's settings, so the Share entry and the details row are for the usual `~/Nextcloud` only. diff --git a/plugins/nextcloud/ask b/plugins/nextcloud/ask index a331acfe..738ed259 100755 --- a/plugins/nextcloud/ask +++ b/plugins/nextcloud/ask @@ -6,8 +6,7 @@ # path asked about. A state is OK, SYNC, NEW, IGNORE, ERROR or NOP (not the # client's), with +SWM after it when shared. Fails when no client is running. -# Without XDG_RUNTIME_DIR the client falls back as Qt does. -socket="${XDG_RUNTIME_DIR:-${TMPDIR:-/tmp}/runtime-$USER}/Nextcloud/socket" +. "${0%/*}/socket.sh" [ -S "$socket" ] || exit 1 # From the plugin's folder, so its path needs no quoting for socat. cd "${0%/*}" || exit 1 diff --git a/plugins/nextcloud/manifest.json b/plugins/nextcloud/manifest.json index 4f728015..d45e8825 100644 --- a/plugins/nextcloud/manifest.json +++ b/plugins/nextcloud/manifest.json @@ -6,5 +6,6 @@ { "id": "share", "label": "Nextcloud…", "menu": "share", "command": ["./share"], "under": ["~/Nextcloud/"], "single": true } ], - "details": { "command": ["./filestatus"], "under": ["~/Nextcloud/"] } + "details": { "command": ["./filestatus"], "under": ["~/Nextcloud/"] }, + "watch": { "command": ["./watch"] } } diff --git a/plugins/nextcloud/socket.sh b/plugins/nextcloud/socket.sh new file mode 100644 index 00000000..832dcc64 --- /dev/null +++ b/plugins/nextcloud/socket.sh @@ -0,0 +1,3 @@ +# Where the Nextcloud client listens, for `ask` and `watch` to read in. +# Without XDG_RUNTIME_DIR the client falls back as Qt does. +socket="${XDG_RUNTIME_DIR:-${TMPDIR:-/tmp}/runtime-$USER}/Nextcloud/socket" diff --git a/plugins/nextcloud/watch b/plugins/nextcloud/watch new file mode 100755 index 00000000..aa5fc4a6 --- /dev/null +++ b/plugins/nextcloud/watch @@ -0,0 +1,7 @@ +#!/bin/sh +# Stays on the Nextcloud client's socket and passes on what it says: a line +# (UPDATE_VIEW) whenever something in a folder it syncs changed, which is +# when Folio should ask again how files stand. It ends when the client isn't +# running, or quits; Folio starts it again. +. "${0%/*}/socket.sh" +exec socat -u "UNIX-CONNECT:$socket" - diff --git a/src/plugins.cpp b/src/plugins.cpp index f5455f92..23ff124b 100644 --- a/src/plugins.cpp +++ b/src/plugins.cpp @@ -161,6 +161,8 @@ Plugins::Plugins(const QString &language, const QStringList &folders, const QStr Plugins::~Plugins() { // Closing Folio ends every command still running, with all it started. + for (const QString &plugin : m_watches.keys()) + endWatch(plugin); *m_stop = true; m_pool.waitForDone(); m_actions.waitForDone(); @@ -278,6 +280,13 @@ std::optional Plugins::parseManifest(const QByteArray &json, m.detailsUnder = *under; m.detailsTypes = *types; } + + if (root.contains("watch")) { + const auto command = commandOf(root.value("watch").toObject().value("command")); + if (!command) + return fail("\"watch\" has no command"); + m.watch = *command; + } return m; } @@ -533,6 +542,43 @@ void Plugins::apply(const QList &manifests) { } // What plugins that went away printed goes with them. m_outputs.removeIf([&](const auto &it) { return !polled.contains(it.key()); }); + + // A watch that's as it was carries on, so F5 or another language doesn't + // start it over; one that couldn't be started gets another go. + QSet watched; + for (const Manifest &m : std::as_const(m_manifests)) { + if (m.watch.isEmpty()) + continue; + watched.insert(m.id); + const auto it = m_watches.constFind(m.id); + if (it != m_watches.constEnd() && it->command == m.watch && it->dir == m.dir + && (it->process || it->restart->isActive())) + continue; + endWatch(m.id); + Watch watch; + watch.command = m.watch; + watch.dir = m.dir; + watch.settle = new QTimer(this); + watch.settle->setSingleShot(true); + watch.settle->setInterval(watchSettle); + connect(watch.settle, &QTimer::timeout, this, [this, id = m.id] { + const auto held = m_watches.find(id); + if (held == m_watches.end() || !held->pending) + return; + held->pending = false; + held->settle->start(); + askAgain(id); + }); + watch.restart = new QTimer(this); + watch.restart->setSingleShot(true); + watch.restart->setInterval(watchRestart); + connect(watch.restart, &QTimer::timeout, this, [this, id = m.id] { startWatch(id); }); + m_watches.insert(m.id, watch); + startWatch(m.id); + } + for (const QString &id : m_watches.keys()) + if (!watched.contains(id)) + endWatch(id); for (const QString &id : polled) if (m_pollAll || !before.contains(id)) poll(id); @@ -569,6 +615,10 @@ void Plugins::poll(const QString &plugin) { connect(watcher, &QFutureWatcherBase::finished, this, [this, watcher, plugin, dir = m->dir, command = m->poll] { watcher->deleteLater(); m_polling.remove(plugin); + // Its watch said something changed meanwhile, which this poll may + // have been too early to see. + if (m_pollAgain.remove(plugin)) + QTimer::singleShot(0, this, [this, plugin] { poll(plugin); }); const Manifest *now = manifest(plugin); if (!now) return; @@ -606,6 +656,123 @@ void Plugins::poll(const QString &plugin) { })); } +void Plugins::startWatch(const QString &plugin) { + const Manifest *m = manifest(plugin); + const auto watch = m_watches.find(plugin); + if (!m || watch == m_watches.end() || watch->process) + return; + // Unlike the other commands this one isn't waited for, so it runs here + // and not in a pool: its lines come in as events. + auto *process = new QProcess(this); + if (!prepare(*process, *m, m->watch, environment(*m))) { + delete process; + return; + } + // A group of its own, so ending it ends everything it started too. + process->setChildProcessModifier([] { ::setpgid(0, 0); }); + // Whether this is still the plugin's watch, and not one ended since. + auto current = [this, plugin, process] { + const auto it = m_watches.find(plugin); + return it != m_watches.end() && it->process == process ? &*it : nullptr; + }; + // What a line says isn't read, only that it came. + connect(process, &QProcess::readyReadStandardOutput, this, [this, plugin, process] { + if (process->readAllStandardOutput().contains('\n')) + noticed(plugin); + }); + // Standard error goes to Folio's, a line at a time, as other commands'. + connect(process, &QProcess::readyReadStandardError, this, [plugin, process, current] { + Watch *watch = current(); + const QList pieces = process->readAllStandardError().split('\n'); + for (qsizetype i = 0; watch && i < pieces.size(); ++i) { + watch->error += pieces[i].left(std::max(0, maxErrorLine - watch->error.size())); + if (i + 1 == pieces.size()) + break; + const QString said = QString::fromUtf8(watch->error).trimmed(); + watch->error.clear(); + if (!said.isEmpty()) + complain(plugin, said); + } + }); + // One that can't be started isn't tried again until the plugins are + // read again; one that ends (what it watches went away) is, in a while. + connect(process, &QProcess::errorOccurred, this, [plugin, process, current](QProcess::ProcessError error) { + Watch *watch = current(); + if (!watch || error != QProcess::FailedToStart) + return; + complain(plugin, process->program() + " couldn't be started: " + process->errorString()); + watch->process = nullptr; + process->deleteLater(); + }); + connect(process, &QProcess::finished, this, [this, plugin, process, current] { + Watch *watch = current(); + if (!watch) + return; + // Whatever it left running in the background goes with it. + if (watch->group > 0) + ::kill(-pid_t(watch->group), SIGKILL); + // Said once each time the watch goes from working to ending. + if (!m_failingWatch.contains(plugin)) + complain(plugin, QString("its watch ended, and is started again every %1 seconds").arg(watchRestart / 1000)); + m_failingWatch.insert(plugin); + watch->process = nullptr; + watch->group = 0; + watch->error.clear(); + process->deleteLater(); + watch->restart->start(); + }); + watch->process = process; + process->start(); + watch->group = process->processId(); +} + +void Plugins::endWatch(const QString &plugin) { + const Watch watch = m_watches.take(plugin); + m_failingWatch.remove(plugin); + delete watch.settle; + delete watch.restart; + if (!watch.process) + return; + watch.process->disconnect(this); + if (watch.group > 0) + ::kill(-pid_t(watch.group), SIGKILL); + watch.process->kill(); + watch.process->waitForFinished(1000); + delete watch.process; +} + +void Plugins::noticed(const QString &plugin) { + const auto watch = m_watches.find(plugin); + if (watch == m_watches.end()) + return; + m_failingWatch.remove(plugin); + // The first line is acted on at once; those that follow close behind + // (a sync going through a folder) are acted on once, when the timer + // rings. + if (watch->settle->isActive()) { + watch->pending = true; + return; + } + watch->settle->start(); + askAgain(plugin); +} + +void Plugins::askAgain(const QString &plugin) { + const Manifest *m = manifest(plugin); + if (!m) + return; + if (!m->details.isEmpty()) { + forgetDetails(plugin); + emit detailsChanged(); + } + // A poll already running may have been too early to see the change, + // so another follows it. + if (m_polling.contains(plugin)) + m_pollAgain.insert(plugin); + else + poll(plugin); +} + QString Plugins::icon(const Manifest &plugin, const QString &name) const { if (!name.endsWith(".svg")) return name.isEmpty() ? QStringLiteral("file") : name; diff --git a/src/plugins.h b/src/plugins.h index c2399fd3..20481466 100644 --- a/src/plugins.h +++ b/src/plugins.h @@ -17,13 +17,16 @@ #include #include +class QProcess; class QTimer; // Plugins: folders holding a manifest.json and commands that print JSON // (docs/plugins.md is the interface). Folio never loads a plugin's code; it // runs its commands in a thread pool of its own, each with a time limit // where it has one, and keeps what they print. A plugin that hangs or -// prints nonsense only goes missing, and the others carry on. +// prints nonsense only goes missing, and the others carry on. The one +// command that stays running is a plugin's watch, which says when something +// changed, so Folio asks again then and not only on a timer. // // What the sidebar, the views and the right-click menu show comes from the // latest output kept here, translated with the plugin's own catalog, so @@ -64,6 +67,9 @@ class Plugins : public QObject { // The command giving rows for the details pane, and the files and // folders it's run for. QStringList details, detailsUnder, detailsTypes; + // The command that stays running and prints a line whenever what + // the plugin says may have changed. + QStringList watch; // The plugin's catalog for the language Folio shows. QHash> strings; }; @@ -183,6 +189,11 @@ class Plugins : public QObject { static constexpr int maxOutput = 1 << 20; // Longest line of standard error kept. static constexpr int maxErrorLine = 1000; + // After a watch's line is acted on, how long the lines that follow are + // held to be acted on once; and how long after a watch ends it's + // started again. + static constexpr int watchSettle = 500; + static constexpr int watchRestart = 5000; public: // A mount finished, so items that mount can find their folder. @@ -191,14 +202,21 @@ class Plugins : public QObject { signals: void changed(); // What plugins say about files may have changed (F5, another language, - // or a plugin's poll saying something new), so the details pane asks - // again. + // a plugin's poll saying something new, or its watch saying so), so the + // details pane asks again. void detailsChanged(); private: void reload(bool pollAll); void apply(const QList &manifests); void poll(const QString &plugin); + // Starts a plugin's watch, and ends it with everything it started. + void startWatch(const QString &plugin); + void endWatch(const QString &plugin); + // A watch printed a line: its plugin is polled and asked about files + // again, once for lines that come close together. + void noticed(const QString &plugin); + void askAgain(const QString &plugin); void rebuild(); const Manifest *manifest(const QString &plugin) const; bool available(const Manifest &plugin, const QString &action) const; @@ -226,6 +244,25 @@ class Plugins : public QObject { // The latest output of each plugin whose poll works. QHash m_outputs; QSet m_polling; + // Plugins whose watch said something changed while their poll ran, so + // they're polled once more when it's done. + QSet m_pollAgain; + // Each plugin's watch: the command and folder it was started with, its + // process while it runs, with the group to end and the line of standard + // error it's in the middle of, the timer holding lines that come close + // together (and whether any came), and the one starting it again. + struct Watch { + QStringList command; + QString dir; + QProcess *process = nullptr; + qint64 group = 0; + QByteArray error; + QTimer *settle = nullptr, *restart = nullptr; + bool pending = false; + }; + QHash m_watches; + // Plugins whose watch ended since it last printed a line. + QSet m_failingWatch; QSet m_failing, m_failingDetails; // What details commands printed, by plugin, path and modification time; // who waits for those still running, the first deciding whether its diff --git a/tests/folio_tests.cpp b/tests/folio_tests.cpp index 940266ea..6ad891ea 100644 --- a/tests/folio_tests.cpp +++ b/tests/folio_tests.cpp @@ -4211,6 +4211,100 @@ private slots: QCOMPARE(asked.readAll().count('\n'), 1); } + void pluginManifestTakesAWatch() + { + QString error; + const auto manifest = Plugins::parseManifest(R"({"folio": 1, "watch": {"command": ["./watch", "-q"]}})", &error); + QVERIFY(manifest); + QCOMPARE(manifest->watch, QStringList({"./watch", "-q"})); + QVERIFY(!Plugins::parseManifest(R"({"folio": 1, "watch": {}})", &error)); + QCOMPARE(error, QString("\"watch\" has no command")); + } + + void pluginWatchHasItPolledAndAskedAgain() + { + // Its watch prints as many lines as `go` says, each time it appears. + QTemporaryDir folder, files; + write(files.filePath("a.txt"), "x"); + writePlugin(folder.path(), "sync", R"({"folio": 1, "poll": {"command": ["./poll"], "every": 86400}, + "details": {"command": ["./describe"]}, "watch": {"command": ["./watch"]}})", + {{"poll", "#!/bin/sh\necho polled >> polls\necho '{}'\n"}, + {"describe", "#!/bin/sh\necho \"[[\\\"State\\\", \\\"$(cat word 2>/dev/null || echo waiting)\\\"]]\"\n"}, + {"watch", "#!/bin/sh\nwhile :; do\n if [ -e go ]; then\n lines=$(cat go); rm go\n" + " while [ \"$lines\" -gt 0 ]; do echo changed; lines=$((lines - 1)); done\n fi\n" + " sleep 0.1\ndone\n"}}); + auto polls = [&] { return contents(folder.filePath("sync/polls")).count('\n'); }; + Plugins plugins("en", {folder.path()}, files.filePath("folio.conf")); + QSignalSpy changed(&plugins, &Plugins::detailsChanged); + QVERIFY(changed.wait(5000)); + QTRY_COMPARE(polls(), 1); + const QString path = files.filePath("a.txt"); + const QDateTime when = QFileInfo(path).lastModified(); + auto describe = [&] { plugins.describe(path, "text/plain", when, [] { return true; }, [] {}); }; + describe(); + QTRY_COMPARE(plugins.details(path, when), QVariantList({QVariantList{"State", "waiting"}})); + + // A line from the watch: the plugin is polled at once, long before + // its timer, and what it said about the file is asked for again. + changed.clear(); + write(folder.filePath("sync/word"), "synced"); + write(folder.filePath("sync/go"), "1"); + QVERIFY(changed.wait(3000)); + QTRY_COMPARE(polls(), 2); + QVERIFY(plugins.details(path, when).isEmpty()); + describe(); + QTRY_COMPARE(plugins.details(path, when), QVariantList({QVariantList{"State", "synced"}})); + + // Lines that come close together are acted on at once and, for + // those that follow the first, once more: never once each. + QTest::qWait(Plugins::watchSettle + 200); + write(folder.filePath("sync/go"), "20"); + QTRY_VERIFY(polls() >= 3); + QTest::qWait(Plugins::watchSettle * 3); + QVERIFY(polls() <= 4); + } + + void pluginWatchIsStartedAgainAndEndsWithItsPlugin() + { + // One watch ends as soon as it starts; the others stay, each with a + // program of its own in the background. + QTemporaryDir folder, files; + const QByteArray stays = "#!/bin/sh\nsleep 600 &\necho \"$$ $!\" >> pids\nwait\n"; + writePlugin(folder.path(), "quits", R"({"folio": 1, "watch": {"command": ["./watch"]}})", + {{"watch", "#!/bin/sh\necho started >> starts\n"}}); + writePlugin(folder.path(), "removed", R"({"folio": 1, "watch": {"command": ["./watch"]}})", {{"watch", stays}}); + writePlugin(folder.path(), "stays", R"({"folio": 1, "watch": {"command": ["./watch"]}})", {{"watch", stays}}); + auto lines = [&](const QString &file) { return contents(folder.filePath(file)).count('\n'); }; + auto gone = [&](const QString &file) { + const QList pids = contents(folder.filePath(file)).trimmed().split(' '); + return pids.size() == 2 && ended(pids[0].toInt()) && ended(pids[1].toInt()); + }; + QByteArray removed; + { + Plugins plugins("en", {folder.path()}, files.filePath("folio.conf")); + QTRY_COMPARE(lines("stays/pids"), 1); + QTRY_COMPARE(lines("removed/pids"), 1); + // A watch that ends is started again in a while. + QTRY_COMPARE_WITH_TIMEOUT(lines("quits/starts"), 2, Plugins::watchRestart + 3000); + + // F5 leaves a running watch be, and ends that of a plugin that + // went away, with what it started. + removed = contents(folder.filePath("removed/pids")); + QDir(folder.filePath("removed")).removeRecursively(); + QSignalSpy read(&plugins, &Plugins::detailsChanged); + plugins.refresh(); + QVERIFY(read.wait(5000)); + QCOMPARE(lines("stays/pids"), 1); + QVERIFY(!gone("stays/pids")); + // What the removed plugin's watch started went with it. + const QList pids = removed.trimmed().split(' '); + QCOMPARE(pids.size(), 2); + QTRY_VERIFY(ended(pids[0].toInt()) && ended(pids[1].toInt())); + } + // Closing Folio ends the rest. + QTRY_VERIFY(gone("stays/pids")); + } + void backendShowsPluginDetailsForTheSelectedItem() { QTemporaryDir folder, home; @@ -4634,6 +4728,36 @@ private slots: client.waitForFinished(); } + void nextcloudPluginWatchesTheClient() + { + const auto plugin = bundledPlugin("nextcloud"); + if (plugin.id.isEmpty()) + QSKIP("socat or jq isn't installed"); + QCOMPARE(plugin.watch, QStringList({"./watch"})); + QTemporaryDir home, dir; + const QString bin = binWithSocat(dir.filePath("bin"), {}); + auto watch = [&](const QString &runtime) { + QProcessEnvironment env; + env.insert("PATH", bin); + env.insert("HOME", home.path()); + env.insert("XDG_RUNTIME_DIR", runtime); + return Plugins::execute(plugin, plugin.watch, env, 1000); + }; + // With no client to listen to it ends, for Folio to start it again. + const Plugins::Run alone = watch(dir.filePath("nothing")); + QVERIFY(alone.started && !alone.timedOut && alone.code != 0); + + // With one it stays, passing on each line the client says. + QProcess client; + QString runtime; + QVERIFY(startNextcloud(client, dir.path(), home.filePath("Nextcloud"), &runtime)); + const Plugins::Run listening = watch(runtime); + QVERIFY(listening.timedOut); + QCOMPARE(listening.out, "REGISTER_PATH:" + home.filePath("Nextcloud").toUtf8() + "\n"); + client.kill(); + client.waitForFinished(); + } + void localSendPluginNeedsLocalSend() { QTemporaryDir dir;