LinkedIn Scraper RESTful API

Overview

This project is a LinkedIn Scraper built using Selenium for web scraping and FastAPI for serving the scraped data via RESTful APIs. It allows users to fetch detailed profile and company information from LinkedIn based on the provided LinkedIn ID.

Installation

Prerequisites

• Python 3.8+
• pip
• git
• Google Chrome

Setup

1. Clone the Repository

   git clone https://github.com/matthewdonsemail-lab/linkedin-scraper-api.git
   cd linkedin-scraper-api

2. Create and Activate Virtual Environment

   python -m venv venv
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install Dependencies

   pip install -r requirements.txt

Configuration

Environment Variables

Create a .env file in the root of the project:

LINKEDIN_ACCESS_TOKEN="YourLinkedInAccessToken"
LINKEDIN_ACCESS_TOKEN_EXP=AccessTokenExpiration
HEADLESS=True

• Replace "YourLinkedInAccessToken" and "AccessTokenExpiration" with your actual LinkedIn access token and its expiration time.
• Change "HEADLESS" to False if you don't want the browser to open in headless mode.

Obtaining LinkedIn Access Token with Cookie-Editor

To fetch data using the LinkedIn Scraper, you'll need a LinkedIn access token. The following steps will guide you through obtaining this token using the Cookie-Editor browser addon:

Step 1: Install Cookie-Editor

Add the Cookie-Editor addon to your browser. It's available for Chrome, Firefox, and other popular browsers.

Step 2: Access LinkedIn

Open your browser and log into your LinkedIn account as you normally would.

Step 3: Open Cookie-Editor

Once logged in, click on the Cookie-Editor icon in your browser's toolbar to open the addon.

Step 4: Find the Access Token

• In the Cookie-Editor interface, look for a cookie named li_at. This cookie contains your LinkedIn access token.
• Click on the li_at cookie to view its details, and copy the value. This is your LinkedIn access token.
• Convert the Expiration time into numeric format (timestamp) like 1743212774.0, and copy the value. This is your LinkedIn access token expiration time.

Notes

• Token Validity: LinkedIn access tokens are temporary. Ensure to check the token's validity periodically and update it as needed.
• Privacy and Security: Handle your access token securely as it grants access to your LinkedIn data. Do not share your token publicly.

Important

If you continue to encounter issues with the access tokens being undefined or expired even after following these steps, please replace the values directly in the settings.py file.

Usage

To start the server, run the following command in the root directory of the project:

python run.py

This command will start the Uvicorn server and make the API accessible on http://localhost:8000 by default.

API Endpoints

The LinkedIn Scraper offers endpoints for retrieving detailed profile and company information from LinkedIn.

Profile Data

• GET /profile-data/{linkedin_id}
  • Fetches profile information for the specified LinkedIn ID.
  • Path Parameters:
    • linkedin_id: The unique LinkedIn ID of the profile.
  • Response Format:

    {
      "linkedin_id": "string",
      "name": "string",
      "headline": "string",
      "education": {
        "positions": ["string"],
        "institutions": ["string"],
        "dates": ["string"]
      },
      "experience": {
        "positions": ["string"],
        "institutions": ["string"],
        "dates": ["string"]
      }
    }

Company Data

• GET /company-data/{linkedin_id}
  • Fetches company information based on the given LinkedIn ID.
  • Path Parameters:
    • linkedin_id: The unique LinkedIn ID of the company.
  • Response Format:

    {
      "linkedin_id": "string",
      "name": "string",
      "industry": "string",
      "about": "string"
    }

Error Handling

Error handling is consistent across endpoints:

{
  "detail": "Error fetching profile details"
}

This response is returned with an HTTP status code of 500, indicating a server-side error during data fetching.

Railway Deployment

This project is configured for deployment on Railway.

Deploy Steps

1. Push this repo to GitHub
2. Create a new project on Railway and connect your GitHub repo
3. Add the following environment variables in Railway dashboard:
   • LINKEDIN_ACCESS_TOKEN - Your LinkedIn li_at cookie value
   • LINKEDIN_ACCESS_TOKEN_EXP - Token expiration timestamp
   • HEADLESS=True
4. Deploy

Railway will automatically:

• Install Google Chrome and ChromeDriver via the included nixpacks.toml
• Use the PORT environment variable
• Start the server with python run.py

Deployment Config Files

• railway.json - Railway deployment configuration
• nixpacks.toml - Installs Chrome/ChromeDriver for Selenium

Project Structure

• api/routes.py - Contains API routes
• services/candidate_scraper.py - Scraping function for individual profiles
• services/company_scraper.py - Scraping function for company profiles
• services/scraping_utils.py - Functions and options used in the scraping process
• settings.py - Configuration settings
• railway.json - Railway deployment config
• nixpacks.toml - Nixpacks build config

Notes

• Ensure that the LinkedIn access token is valid and not expired to avoid authentication errors.
• The scraper requires Google Chrome to be installed (handled automatically on Railway).