Skip to content

Commit 7b1319b

Browse files
devilscrapesdevilscrapes
committed
GitHub user scraper for Python
0 parents  commit 7b1319b

12 files changed

Lines changed: 569 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.11", "3.13"]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: pytest -q

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.env
2+
results.json
3+
results.csv
4+
__pycache__/
5+
.venv/
6+
.pytest_cache/
7+
.ruff_cache/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/github-user-scraper"><img src="assets/banner.svg" alt="GitHub User Scraper — Profiles, Repos &amp; Orgs — Python example" width="100%"></a></p>
2+
3+
# GitHub User Scraper in Python: profiles and organizations to pandas, JSON and CSV
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/github-user-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="Python" src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/github-user-scraper-python/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/github-user-scraper-python/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
This repo is a tested Python starter for the **[GitHub User Scraper](https://apify.com/DevilScrapes/github-user-scraper)** on Apify. You give it account names, for example `apify` and `microsoft`, and it hands back one row per account: display name, kind of account, follower and following totals, number of public repositories and gists, declared location, company and website, creation date and, when you switch it on, a short list of the repositories that changed most recently. The lookups run on Apify, so your notebook or cron job only waits for the run and reads `results.json` or `results.csv`. It is the kind of table you would otherwise assemble by clicking through profiles one by one, which is a special kind of purgatory we would rather automate.
13+
14+
> **Actor:** [GitHub User Scraper — Profiles, Repos & Orgs](https://apify.com/DevilScrapes/github-user-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in JavaScript:** [github-user-scraper-javascript](https://github.com/devilscrapes/github-user-scraper-javascript)
15+
16+
## 🔥 What you get
17+
18+
- **People and organizations in one schema**: both kinds of account come back with identical keys, and `type` says `User` or `Organization` so you can split them later.
19+
- **Paste logins or links**: entries in `usernames` may be bare slugs or full profile URLs; the host part is removed before lookup.
20+
- **Counts as integers**: `followers`, `following`, `public_repos` and `public_gists` are numbers, ready to sort, sum or chart.
21+
- **Recent repositories on request**: set `includeRepos` and each row gains a `repos` list of the account's most recently updated public repositories, each with name, description, language, stars, forks, fork and archived flags and last push time. `maxReposPerUser` caps it at up to 100.
22+
- **Bring a token if you need volume**: without one, GitHub allows 60 requests an hour; an optional personal access token lifts that to 5,000.
23+
- **Back-off built in**: on throttling or server errors the Actor waits and retries, and it reports a clear status message instead of finishing with a silently empty dataset.
24+
- **Only what is public**: `email` is filled only when the account owner chose to publish one, and private repositories or memberships are never touched.
25+
26+
## 🎯 Use cases
27+
28+
### Benchmark open-source presence across companies
29+
30+
List the GitHub organizations of a set of vendors, load the rows into a DataFrame and compare `followers` and `public_repos` side by side. Repeat it each quarter from a scheduled job and you have a simple trend line of developer reach.
31+
32+
### Find out what an organization actually works on
33+
34+
Turn on `includeRepos`, then count the `language` values and sum `stargazers_count` across the repositories you got back. A notebook cell tells you whether a company's recent public work is mostly TypeScript, Go or Python.
35+
36+
### Clean up a list of GitHub links in a spreadsheet
37+
38+
Export the profile URLs from a CRM or conference sheet, pass them straight in as `usernames` and merge the result back on `login`. Accounts that no longer resolve stand out because they are missing from the output.
39+
40+
### Track developer-relations metrics over time
41+
42+
Snapshot your own organization and a few peers daily with cron, write each CSV to a dated folder and plot follower growth with matplotlib. Because `scraped_at` is on every row, concatenating snapshots needs no extra bookkeeping.
43+
44+
## ⚡ Quick start
45+
46+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
47+
48+
```bash
49+
git clone https://github.com/devilscrapes/github-user-scraper-python.git
50+
cd github-user-scraper-python
51+
python -m venv .venv && source .venv/bin/activate
52+
pip install -r requirements.txt
53+
export APIFY_TOKEN=your_token_here
54+
python main.py # uses input.json
55+
python main.py my-input.json --out data --max-charge 2
56+
```
57+
58+
The script starts the [GitHub User Scraper — Profiles, Repos & Orgs](https://apify.com/DevilScrapes/github-user-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
59+
60+
## 🧩 The core call
61+
62+
Underneath the CLI wrapper, the integration fits in a few lines. The official `apify-client` package starts the run, blocks until it is done and iterates over the dataset for you.
63+
64+
```python
65+
from apify_client import ApifyClient
66+
67+
client = ApifyClient("YOUR_APIFY_TOKEN")
68+
run = client.actor("DevilScrapes/github-user-scraper").call(run_input={'usernames': ['apify', 'microsoft'], 'includeRepos': True, 'maxReposPerUser': 5, 'proxyConfiguration': {'useApifyProxy': False}})
69+
for item in client.dataset(run.default_dataset_id).iterate_items():
70+
print(item)
71+
```
72+
73+
## 📥 Input
74+
75+
The script ships `input.json` to the Actor without changes. The example looks up two organizations and asks for the five most recently updated repositories of each, so the run returns two rows and finishes quickly. `usernames` is the only required field. Add a `githubToken` in a private input file, never in a committed one, once your list grows past a few dozen accounts.
76+
77+
```json
78+
{
79+
"usernames": [
80+
"apify",
81+
"microsoft"
82+
],
83+
"includeRepos": true,
84+
"maxReposPerUser": 5,
85+
"proxyConfiguration": {
86+
"useApifyProxy": false
87+
}
88+
}
89+
```
90+
91+
| Field | Type | Required | Default | Allowed values |
92+
|---|---|---|---|---|
93+
| `usernames` | array | yes | `["torvalds", "apify"]` | — |
94+
| `githubToken` | string | no | — | — |
95+
| `includeRepos` | boolean | no | `false` | — |
96+
| `maxReposPerUser` | integer | no | `30` | — |
97+
| `concurrency` | integer | no | `6` | — |
98+
| `proxyConfiguration` | object | no | `{"useApifyProxy": false}` | — |
99+
100+
## 📤 Output
101+
102+
You get one record per account. `login`, `type` and `html_url` are guaranteed; profile fields that the owner left blank arrive as null. `repos` is null unless `includeRepos` was true, and in that case it is a list of small dictionaries rather than a flat column, which is why the recipe below unpacks it.
103+
104+
| Field | Type | Always present |
105+
|---|---|---|
106+
| `login` | string | yes |
107+
| `type` | string | yes |
108+
| `name` | string or null | no |
109+
| `company` | string or null | no |
110+
| `blog` | string or null | no |
111+
| `location` | string or null | no |
112+
| `email` | string or null | no |
113+
| `bio` | string or null | no |
114+
| `twitter_username` | string or null | no |
115+
| `public_repos` | integer | no |
116+
| `public_gists` | integer | no |
117+
| `followers` | integer | no |
118+
| `following` | integer | no |
119+
| `html_url` | string | yes |
120+
| `avatar_url` | string | no |
121+
| `hireable` | boolean or null | no |
122+
| `created_at` | string | no |
123+
| `updated_at` | string | no |
124+
| `repos` | array or null | no |
125+
| `scraped_at` | string | no |
126+
127+
## 💰 Pricing
128+
129+
| Event | Price (USD) |
130+
|---|---|
131+
| Actor start | $0.2 |
132+
| Result emitted | $0.003 |
133+
134+
Prices as configured on 2026-08-20. The [Actor page](https://apify.com/DevilScrapes/github-user-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
135+
136+
## 🛠️ Working with the data in Python
137+
138+
This pandas snippet (`pip install pandas`) builds an account summary: it computes how long each account has existed, unpacks the nested `repos` lists, totals stars per account and picks the most common language, then saves `accounts.csv`.
139+
140+
```python
141+
import json
142+
143+
import pandas as pd
144+
145+
accounts = json.load(open("results.json", encoding="utf-8"))
146+
profiles = pd.DataFrame(accounts)[["login", "type", "followers", "public_repos", "created_at"]]
147+
created = pd.to_datetime(profiles["created_at"], utc=True)
148+
profiles["years_on_github"] = ((pd.Timestamp.now(tz="UTC") - created).dt.days / 365.25).round(1)
149+
150+
repo_rows = [{**repo, "owner": acc["login"]} for acc in accounts for repo in (acc.get("repos") or [])]
151+
repos = pd.DataFrame(repo_rows)
152+
if not repos.empty:
153+
per_owner = repos.groupby("owner").agg(
154+
sampled_repos=("name", "count"),
155+
stars=("stargazers_count", "sum"),
156+
top_language=("language", lambda s: s.mode().iat[0] if not s.mode().empty else None),
157+
)
158+
profiles = profiles.join(per_owner, on="login")
159+
160+
profiles.to_csv("accounts.csv", index=False)
161+
print(profiles.to_string(index=False))
162+
```
163+
164+
## ❓ FAQ
165+
166+
### Do I need a GitHub token?
167+
168+
Not for small lists. Without a token GitHub allows 60 requests per hour from one address, and every account costs one request, plus one more when `includeRepos` is on. For hundreds of accounts, put a personal access token in `githubToken` and the allowance rises to 5,000 an hour.
169+
170+
### Why is `email` almost always null?
171+
172+
GitHub hides addresses unless the account owner chose to make one public. The Actor returns the public field as it is and does not try to recover addresses from commits or anywhere else.
173+
174+
### Can I get contribution graphs or commit counts?
175+
176+
No. The data comes from the public profile, not from activity history, so contribution calendars, sponsorship tiers, starred repositories and commit statistics are not part of a row. For repository-level detail, look at the related repo scraper below.
177+
178+
### Is it fine to use this data for recruiting outreach?
179+
180+
The Actor only returns fields people chose to publish, such as the self-declared `hireable` flag. How you use them, and whether that use meets GDPR and GitHub's acceptable use rules, is your responsibility. Keep personal data to what you actually need.
181+
182+
## 🧯 Troubleshooting
183+
184+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
185+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
186+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
187+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/github-user-scraper) so the maintainers see it.
188+
189+
## 🔗 More from Devil Scrapes
190+
191+
- [GitHub User Scraper — Profiles, Repos & Orgs on Apify Store](https://apify.com/DevilScrapes/github-user-scraper)
192+
- [JavaScript version of this example](https://github.com/devilscrapes/github-user-scraper-javascript)
193+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
194+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
195+
196+
Related Actors:
197+
198+
- [GitHub Organization Scraper](https://apify.com/DevilScrapes/github-org-scraper) — organization metadata plus its public repository list, including member counts.
199+
- [GitHub Repo Scraper](https://apify.com/DevilScrapes/github-repo-scraper) — deep metadata for individual repositories, such as topics, license and latest release.
200+
- [GitHub Trending Scraper](https://apify.com/DevilScrapes/github-trending-scraper) — what is climbing GitHub's trending page today.
201+
202+
---
203+
204+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/github-user-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with GitHub or Microsoft.

‎assets/banner.svg‎

Lines changed: 41 additions & 0 deletions
Loading

‎input.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"usernames": [
3+
"apify",
4+
"microsoft"
5+
],
6+
"includeRepos": true,
7+
"maxReposPerUser": 5,
8+
"proxyConfiguration": {
9+
"useApifyProxy": false
10+
}
11+
}

0 commit comments

Comments
 (0)