Skip to content

Latest commit

 

History

35 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

qURL Proxy & qURL Artisan

qURL Proxy is a non-caching, Node.js HTTP proxy that supports batch requests and is driven by URL query. Headers, methods, bodies, and status codes can be overridden, and headers can also be deleted using wildcards. Responses can be transformed through custom JavaScript logic, which can chain requests and merge responses. It also supports retries with exponential backoff, timeouts, throttling and optional limits on request batching and recursion. By default it strips sensitive request headers and bypasses CORS response restrictions, useful for debugging and development. Notes · Examples

Usage

Server

  • Public instance - clone
  • Local instance - npm start
  • CLI instance (no client) - npx -y qurl-proxy

Library

import {createProxy} from 'qurl-proxy'
const proxy = createProxy(config)
const response = await proxy(request)

URL Parameters

  • key - API secret, if required
  • url - resource URL, http assumed, required, repeatable (max. 16), first response used, other statuses in comma-separated X-Proxy-Responses
  • fastest - return first available response and its index in X-Proxy-Responses, abort others
  • headers - JSON or JSONCrush object of request headers to overwrite (Host is determined dynamically)
  • delheaders - JSON or JSONCrush array of names of request headers to delete (Connection is deleted along with headers listed in it, * is a wildcard), in addition to:
    [
      // https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers#hop-by-hop_headers
      "Connection", "Keep-Alive", "Proxy-Authorization", "Trailer", "Transfer-Encoding", "TE", "Upgrade",
      // https://developer.mozilla.org/docs/Web/HTTP/Reference/Status/304
      "Cache-Control", "Pragma", "If-Modified-Since", "If-None-Match",
      // real addresses
      "Origin", "Referer", "Via", "Forwarded", "X-Forwarded-*", "*-IP",
      // browser data
      "Sec-CH-*", "Sec-Fetch-*",
    ]
  • resheaders - JSON or JSONCrush object of response headers to overwrite (Access-Control-Allow-Origin and Access-Control-Expose-Headers are set automatically), in addition to:
    {
      "Access-Control-Allow-Headers": "*",
      "Access-Control-Allow-Credentials": "true",
      "Cross-Origin-Resource-Policy": "cross-origin",
      "Timing-Allow-Origin": "*"
    }
  • delresheaders - JSON or JSONCrush array of names of response headers to delete (Connection is deleted along with headers listed in it, * is a wildcard), in addition to:
    [
      // https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers#hop-by-hop_headers
      "Connection", "Keep-Alive", "Proxy-Authenticate", "Trailer", "Transfer-Encoding", "Upgrade",
    ]
  • renresheaders - rename response headers to X-Original-* before changes
  • skipdefaults - do not apply default header changes, except response safety behavior and setting response X-Proxy-Recursion (max. 16)
  • method - request method override
  • body - request body text
  • resbody - response transformation:
    • null - remove response body
    • atob - decode body from Base64
    • btoa - encode body to Base64
    • javascript:… - custom handler, returns body, response or request
  • status - response status code to overwrite
  • retry - retries after first request
  • retryin - milliseconds between retries, supports exponential backoff:
    min(in * factorattempt, limit)
  • retryfactor - backoff multiplier per retry (default is 1, industry standard is 2)
  • retrylimit - backoff maximum milliseconds
  • timeout - milliseconds to abort request after (default is 300000)
  • ttfb - milliseconds to first response byte
  • throttle - bidirectional bandwidth limit in kbit/s
  • throttleup - upload bandwidth limit in kbit/s

Response Headers Safety

// https://github.com/nodejs/undici/issues/2514
if (headers.get('Content-Encoding')) {
  headers.delete('Content-Encoding')
  headers.delete('Content-Length')
}
// recompress
const contentEncoding = resolveAcceptHeader(headers.get('Accept-Encoding')) || 'gzip'
if (contentEncoding !== 'identity') {
  headers.set('Content-Encoding', contentEncoding)
  headers.delete('Content-Length')
  headers.set('Transfer-Encoding', 'chunked')
}
// resbody param
if (['null', 'atob', 'btoa'].includes(params.get('resbody')?.toLowerCase()))
  headers.delete('Content-Length')

After running resbody custom handler

if (!result instanceof Request && !result instanceof Response && result !== undefined)
  headers.delete('Content-Length')

TypeScript Declaration of resbody=javascript:…

declare function custom(
  // request with parameters applied
  req: RequestView,
  // first or fastest response with parameters applied
  res: ResponseView,
  // other responses, null if error
  responses: Array<ResponseView | null>
): CustomResult

interface ReqResView {
  url: string
  headers: Record<string, string>
  // body:
  body: ReadableStream | null
  bytes: Uint8Array
  text: string
  json: any
}

interface RequestView extends ReqResView {
  method: string
}

interface ResponseView extends ReqResView {
  cookies: string[]
  ok: boolean
  redirected: boolean
  status: number
  statusText: string
}

type CustomResult =
  | Request                     // replace original request and refetch response
  | Response                    // replace original response
  | undefined                   // return original response
  | ReadableStream | Uint8Array // replace response body with value
  | unknown                     // replace response body with coerced value?.toString()
  | null                        // remove response body

Notes Ask DeepWiki

License FOSSA Status

Licensed under the Apache License, Version 2.0. See:

(Top)

Releases

Contributors

Languages