Skip to content

Commit e070ffa

Browse files
authored
Merge pull request #16 from ecency/canonical-and-app-registry-refresh
Ecency canonical to /@user/permlink, refresh app registry, validate data
2 parents fc0d2c4 + e70af8f commit e070ffa

13 files changed

Lines changed: 10817 additions & 137 deletions

.github/workflows/release.yml

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
name: Publish to npm
2+
23
on:
34
push:
45
branches:
@@ -8,11 +9,13 @@ jobs:
89
publish:
910
runs-on: ubuntu-latest
1011
steps:
11-
- uses: actions/checkout@v1
12-
- uses: actions/setup-node@v1
12+
- uses: actions/checkout@v4
13+
- uses: actions/setup-node@v4
1314
with:
14-
node-version: 12
15-
- run: yarn
16-
- uses: JS-DevTools/npm-publish@v1
15+
node-version: 20
16+
registry-url: https://registry.npmjs.org
17+
# Never publish data that would not pass review.
18+
- run: node scripts/validate.mjs
19+
- uses: JS-DevTools/npm-publish@v3
1720
with:
1821
token: ${{ secrets.NPM_TOKEN }}

.github/workflows/validate.yml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
name: Validate data
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- master
8+
9+
jobs:
10+
validate:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- uses: actions/setup-node@v4
15+
with:
16+
node-version: 20
17+
# No dependencies to install: the validator uses node builtins only.
18+
- run: node scripts/validate.mjs

README.md

Lines changed: 123 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -5,75 +5,167 @@ An open standard for Hive based apps.
55
- `Apps` - URL format and canonical linking schemes
66
- `BadActors` - accounts mischiefs or phishing attempts
77
- `BadDomains` - phishing domains
8+
- `GoodDomains` - domains known to be safe
9+
- `Spaminator` - larger imported lists maintained by the Spaminator project
810

911
# How to use this package
1012

1113
`yarn add @hiveio/hivescript`
1214

13-
## Canonical linking
15+
## Files
1416

15-
On Hive, content is stored in blockchain and same information is accessible via different websites and services built on Hive. Canonical linking to origin of post is important for entire ecosystem to thrive.
17+
| File | Shape | What it is |
18+
| --- | --- | --- |
19+
| `apps.json` | object | App registry: display name, homepage and canonical `url_scheme` |
20+
| `bad-actors.json` | string[] | Accounts reported for phishing / typosquatting exchange names |
21+
| `bad-domains.json` | string[] | Phishing domains, curated |
22+
| `good-domains.json` | string[] | Domains known to be safe |
23+
| `spaminator-domains.json` | string[] | Domain blocklist imported from Spaminator |
24+
| `spaminator-all.json` | string[] | Full Spaminator account blocklist (~174k entries, 2 MB) |
1625

17-
Here is an example on how to do it in few simple lines:
26+
`spaminator-all.json` is large. Import it only if you actually need it, and never
27+
into a browser bundle.
1828

19-
```
20-
import apps from "@hiveio/hivescript/apps.json";
29+
## Canonical linking
2130

22-
let scheme = `${default_domain}/{category}/@{username}/{permlink}`;
31+
On Hive, content is stored in blockchain and same information is accessible via different
32+
websites and services built on Hive. Canonical linking to origin of post is important for
33+
entire ecosystem to thrive.
2334

24-
// get app information from post json
25-
const app = post.json_metadata.app;
35+
Two things about `apps.json` decide the shape of the code below:
2636

27-
if (app) {
28-
const identifier = app.split("/")[0];
37+
- **`url_scheme` is optional.** Some entries are publishing tools with no web home of their
38+
own (`beempy`, `steempress`). Reading `.url_scheme` off those gives `undefined`, so always
39+
fall back to your own scheme rather than assuming it is there.
40+
- **Not every scheme uses `{category}`.** A scheme may contain `{category}`, `{username}` and
41+
`{permlink}` in any combination. Replace whatever is present and leave the rest alone.
2942

30-
if (apps[identifier]) {
31-
scheme = apps[identifier].url_scheme;
43+
```js
44+
import apps from "@hiveio/hivescript/apps.json";
45+
46+
// Your own site's scheme, used whenever the post's app is unknown to us.
47+
const DEFAULT_SCHEME = "https://example.com/{category}/@{username}/{permlink}";
48+
49+
function canonicalLink(entry, defaultScheme = DEFAULT_SCHEME) {
50+
// json_metadata is an object on bridge.* but a JSON string on condenser_api.*
51+
let meta = entry.json_metadata;
52+
if (typeof meta === "string") {
53+
try {
54+
meta = JSON.parse(meta);
55+
} catch {
56+
meta = {};
3257
}
3358
}
34-
// return proper canonical link for post
35-
const canonicalLink = scheme
59+
60+
// `app` is normally "ecency/3.1.4" but some apps write an object instead. Neither form
61+
// is guaranteed: json_metadata is arbitrary author-supplied JSON, so check the type
62+
// before calling string methods on it.
63+
const app = meta?.app;
64+
const raw = typeof app === "string" ? app : app?.name;
65+
const identifier = typeof raw === "string" ? raw.split("/")[0].trim().toLowerCase() : undefined;
66+
67+
// Falls back when the app is unknown OR known but has no url_scheme of its own.
68+
const scheme = (identifier && apps[identifier]?.url_scheme) || defaultScheme;
69+
70+
return scheme
3671
.replace("{category}", entry.category)
3772
.replace("{username}", entry.author)
3873
.replace("{permlink}", entry.permlink);
39-
74+
}
4075
```
4176
77+
### Contributing
78+
79+
`node scripts/validate.mjs` checks every data file: shape, sorting, duplicates, casing,
80+
good/bad overlap, public suffixes and `apps.json` placeholders. CI runs it on every pull
81+
request and again before publish. No dependencies to install.
82+
83+
The public suffix check reads `scripts/public-suffix-list.txt`, a snapshot of the
84+
[Public Suffix List](https://publicsuffix.org/list/) refreshed by
85+
`node scripts/update-public-suffix-list.mjs`. That snapshot is MPL 2.0, carries its upstream
86+
notice, and is development tooling only: it is outside the `files` allowlist, so the npm
87+
package stays MIT.
88+
89+
### Adding or changing an app
90+
91+
Open a pull request against `apps.json`. Entries are sorted by key. A `url_scheme` must be
92+
`https`, must contain `{permlink}`, and must resolve to a real post page: no hash fragments
93+
(`#!/...`), because search engines do not treat those as distinct canonical URLs. Entries
94+
whose domain stops resolving, starts redirecting off-site or gets parked are removed, since a
95+
stale entry sends every frontend's canonical links and the SEO authority behind them to
96+
whoever holds the domain now.
97+
4298
## Bad actors
4399
44-
Bad actors, list of account that is mostly created with intention to take advantage of user mistype. Sometimes simple misspell can direct funds into wrong accounts, this list contain those reported accounts.
100+
Bad actors, list of account that is mostly created with intention to take advantage of user
101+
mistype. Sometimes simple misspell can direct funds into wrong accounts, this list contain
102+
those reported accounts.
45103
46-
This section could be part of wallet page in your Dapp where user enters account name to transfer funds to.
104+
This section could be part of wallet page in your Dapp where user enters account name to
105+
transfer funds to.
47106
48-
```
49-
import badActors from '@hiveio/hivescript/bad-actors.json';
107+
Build a `Set` once at module load. The list is over a thousand entries and `Array.includes`
108+
re-scans all of it on every keystroke.
50109
51-
if (badActors.includes(to_account)) {
52-
console.warn("Use caution sending to this account. Please double check your spelling for possible phishing.");
53-
}
110+
```js
111+
import badActors from "@hiveio/hivescript/bad-actors.json";
54112

55-
```
113+
const BAD_ACTORS = new Set(badActors);
56114

115+
// Hive account names are lowercase; normalise before comparing.
116+
if (BAD_ACTORS.has(to_account.trim().toLowerCase().replace(/^@/, ""))) {
117+
console.warn(
118+
"Use caution sending to this account. Please double check your spelling for possible phishing."
119+
);
120+
}
121+
```
57122
58123
## Bad domains
59124
60-
Phishing domains, list of phishing domains, we recommend Dapp/frontend developers check external link clicks and warn users about potential phishing domains.
125+
Phishing domains, list of phishing domains, we recommend Dapp/frontend developers check
126+
external link clicks and warn users about potential phishing domains.
61127
62-
This section could be part of content rendering or external link clicking event listener in your web/mobile/desktop apps.
128+
This section could be part of content rendering or external link clicking event listener in
129+
your web/mobile/desktop apps.
63130
64-
```
65-
import badDomains from '@hiveio/hivescript/bad-domains.json';
131+
Parse the URL rather than matching it with a regex. `new URL()` lowercases the host and
132+
converts internationalised domains to punycode, which is what the list stores, so homograph
133+
domains such as `șteemit.com` (`xn--teemit-2lc.com`) are caught. Then walk the parent domains,
134+
otherwise `login.phishing-site.tk` slips past an entry for `phishing-site.tk`.
135+
136+
Because consumers walk parent domains, every entry in these lists has to be a registrable
137+
domain. A public suffix such as `web.app`, `github.io` or `co.uk` would condemn every site
138+
hosted under it, so list the specific abusive hostname instead. CI rejects entries that are
139+
public suffixes.
66140
67-
const regex = /^(?:https?:\/\/)?(?:[^@\/\n]+@)?(?:www\.)?([^:\/?\n]+)/
141+
```js
142+
import badDomains from "@hiveio/hivescript/bad-domains.json";
68143

69-
external_link = external_link.match(regex)[1]
144+
const BAD_DOMAINS = new Set(badDomains);
70145

71-
if (badDomains.includes(external_link)) {
72-
console.warn("Security alert! Site ahead contains malware / Suspected phishing page.");
146+
function isBadDomain(externalLink) {
147+
let host;
148+
try {
149+
// A terminal dot is a valid, fully qualified host: browsers resolve
150+
// "steemit24.cf." exactly like "steemit24.cf", so strip it before matching.
151+
host = new URL(externalLink).hostname.toLowerCase().replace(/\.$/, "").replace(/^www\./, "");
152+
} catch {
153+
return false; // not a URL we can judge
73154
}
74155

156+
// "a.b.evil.tk" -> checks "a.b.evil.tk", "b.evil.tk", "evil.tk"
157+
const labels = host.split(".");
158+
return labels.some((_, i) => BAD_DOMAINS.has(labels.slice(i).join(".")));
159+
}
160+
161+
if (isBadDomain(external_link)) {
162+
console.warn("Security alert! Site ahead contains malware / Suspected phishing page.");
163+
}
75164
```
76165
166+
`new URL()` needs an absolute URL. If you are checking hrefs straight out of post bodies,
167+
resolve them first: `new URL(href, "https://example.com")`.
168+
77169
# Contributors
78170
79171
[Hive community](https://hive.io)

apps.json

Lines changed: 60 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -1,95 +1,90 @@
11
{
2-
"hiveblog": {
3-
"name": "Hive blog",
4-
"homepage": "https://hive.blog",
5-
"url_scheme": "https://hive.blog/{category}/@{username}/{permlink}"
6-
},
7-
"peakd": {
8-
"name": "PeakD",
9-
"homepage": "https://peakd.com",
10-
"url_scheme": "https://peakd.com/{category}/@{username}/{permlink}"
11-
},
12-
"ecency": {
13-
"name": "Ecency",
14-
"homepage": "https://ecency.com",
15-
"url_scheme": "https://ecency.com/{category}/@{username}/{permlink}"
2+
"3speak": {
3+
"name": "3Speak",
4+
"homepage": "https://3speak.tv",
5+
"url_scheme": "https://3speak.tv/watch?v={username}/{permlink}"
166
},
177
"actifit": {
188
"name": "Actifit",
199
"homepage": "https://actifit.io",
2010
"url_scheme": "https://actifit.io/@{username}/{permlink}"
2111
},
22-
"3speak": {
23-
"name": "3Speak",
24-
"homepage": "https://3speak.tv",
25-
"url_scheme": "https://3speak.tv/watch?v={username}/{permlink}"
12+
"beempy": {
13+
"name": "beempy",
14+
"homepage": "https://github.com/holgern/beem"
2615
},
27-
"stemsocial": {
28-
"name": "STEMsocial",
29-
"homepage": "https://stem.openhive.network",
30-
"url_scheme": "https://stem.openhive.network/#!/@{username}/{permlink}"
16+
"dtube": {
17+
"name": "DTube",
18+
"homepage": "https://d.tube",
19+
"url_scheme": "https://d.tube/v/{username}/{permlink}"
3120
},
32-
"leofinance": {
33-
"name": "Leo Finance",
34-
"homepage": "https://leofinance.io",
35-
"url_scheme": "https://leofinance.io/{category}/@{username}/{permlink}"
21+
"ecency": {
22+
"name": "Ecency",
23+
"homepage": "https://ecency.com",
24+
"url_scheme": "https://ecency.com/@{username}/{permlink}"
3625
},
3726
"esteem": {
3827
"name": "Esteem",
3928
"homepage": "https://ecency.com",
40-
"url_scheme": "https://ecency.com/{category}/@{username}/{permlink}"
29+
"url_scheme": "https://ecency.com/@{username}/{permlink}"
4130
},
42-
"steempress": {
43-
"name": "SteemPress",
44-
"homepage": "https://wordpress.org/plugins/steempress/"
31+
"hiveblog": {
32+
"name": "Hive blog",
33+
"homepage": "https://hive.blog",
34+
"url_scheme": "https://hive.blog/{category}/@{username}/{permlink}"
4535
},
46-
"beempy": {
47-
"name": "beempy",
48-
"homepage": "https://github.com/holgern/beem"
36+
"leofinance": {
37+
"name": "InLeo",
38+
"homepage": "https://inleo.io",
39+
"url_scheme": "https://inleo.io/@{username}/{permlink}"
4940
},
50-
"travelfeed": {
51-
"name": "TravelFeed",
52-
"homepage": "https://travelfeed.com",
53-
"url_scheme": "https://travelfeed.com/@{username}/{permlink}"
41+
"leothreads": {
42+
"name": "InLeo Threads",
43+
"homepage": "https://inleo.io",
44+
"url_scheme": "https://inleo.io/@{username}/{permlink}"
5445
},
55-
"clicktrackprofit": {
56-
"name": "ClickTrackProfit",
57-
"homepage": "https://www.ctptalk.com",
58-
"url_scheme": "https://www.ctptalk.com/{category}/@{username}/{permlink}"
46+
"liketu": {
47+
"name": "Liketu",
48+
"homepage": "https://liketu.com",
49+
"url_scheme": "https://liketu.com/post/{username}/{permlink}"
5950
},
60-
"dtube": {
61-
"name": "DTube",
62-
"homepage": "https://d.tube",
63-
"url_scheme": "https://d.tube/v/{username}/{permlink}"
51+
"peakd": {
52+
"name": "PeakD",
53+
"homepage": "https://peakd.com",
54+
"url_scheme": "https://peakd.com/{category}/@{username}/{permlink}"
6455
},
65-
"steemit": {
66-
"name": "Steemit",
67-
"homepage": "https://steemit.com",
68-
"url_scheme": "https://steemit.com/{category}/@{username}/{permlink}"
56+
"propolis.eng": {
57+
"name": "propolis.eng",
58+
"homepage": "https://propol.is",
59+
"url_scheme": "https://propol.is/wiki/{permlink}"
6960
},
70-
"inji": {
71-
"name": "inji",
72-
"homepage": "https://inji.com",
73-
"url_scheme": "https://inji.com/hive/@{username}/{permlink}"
61+
"scrobble.life": {
62+
"name": "Scrobble.life",
63+
"homepage": "https://scrobble.life",
64+
"url_scheme": "https://scrobble.life/p/{username}/{permlink}"
7465
},
7566
"splintertalk": {
7667
"name": "splintertalk",
7768
"homepage": "https://www.splintertalk.io",
7869
"url_scheme": "https://www.splintertalk.io/@{username}/{permlink}"
7970
},
80-
"proofofbrain": {
81-
"name": "proofofbrain",
82-
"homepage": "https://proofofbrain.io",
83-
"url_scheme": "https://proofofbrain.io/{category}/@{username}/{permlink}"
71+
"steemit": {
72+
"name": "Steemit",
73+
"homepage": "https://steemit.com",
74+
"url_scheme": "https://steemit.com/{category}/@{username}/{permlink}"
75+
},
76+
"steempress": {
77+
"name": "SteemPress",
78+
"homepage": "https://wordpress.org/plugins/steempress/"
8479
},
85-
"eskateraleigh": {
86-
"name": "eskateraleigh",
87-
"homepage": "https://eskateraleigh.com",
88-
"url_scheme": "https://eskateraleigh.com/blogPost/@{username}/{permlink}"
80+
"travelfeed": {
81+
"name": "TravelFeed",
82+
"homepage": "https://travelfeed.com",
83+
"url_scheme": "https://travelfeed.com/@{username}/{permlink}"
8984
},
90-
"propolis.eng": {
91-
"name": "propolis.eng",
92-
"homepage": "https://propol.is",
93-
"url_scheme": "https://propol.is/wiki/{permlink}"
85+
"waivio": {
86+
"name": "Waivio",
87+
"homepage": "https://www.waivio.com",
88+
"url_scheme": "https://www.waivio.com/@{username}/{permlink}"
9489
}
9590
}

0 commit comments

Comments
 (0)