Can I get INDENT and DEDENT tokens?

[leodevbro]

With the word "token" I mean an info of any INDENT/DEDENT about the line/column location in the text.

Hello, I'm working on my VSCode extension Blockman. Blockman uses many parsing libraries to analyze text of many programming languages.

For python language, it uses dt-python-parser to find all the INDENT and DEDENT tokens. it works pretty well but very slow.

This is how it works:

const classPython3Parser = require("dt-python-parser").Python3Parser;

const pyParser = new classPython3Parser();

const typeS = 93; // type 93 is INDENT
const typeE = 94; // type 94 is DEDENT


const pyTokens: any[] = pyParser.getAllTokens(pythonText);


let indentsAndDedents: {
    type: number;
    line: number;
    column: number;
    start: number;
    stop: number;
}[] = [];

pyTokens.map((x: any) => {
    if ([typeS, typeE].includes(x.type)) { // filtering only INDENT and DEDENT tokens. I don't need other tokens
        indentsAndDedents.push({
            type: x.type,
            line: x.line,
            column: x.column,
            start: x.start,
            stop: x.stop,
        });
    }
});

getAllTokens function is very slow, maybe because it parses everything, not just INDENTs and DEDENTs. It takes about 10 seconds to parse 10,000 line file. It is too slow.

Can I somehow optimize finding INDENT/DEDENT tokens? Maybe with python-indent-parser? It seems to be focused only on indentations and nothing more, so maybe it will be faster, but I don't know how to get all indentation tokens with python-indent-parser. Or maybe there is another good parser of indentation? Please help.