Introduction

pyadql -- ADQL 2.1 parser producing a typed Python AST.

Example:

>>> from pyadql import parse_adql
>>> tree = parse_adql("SELECT TOP 10 ra, dec FROM mytable WHERE ra > 10")
>>> tree.body.top
10

Note the root node is a SelectExpression, not the SELECT itself: per ADQL 2.1's own grammar, ORDER BY/OFFSET/WITH apply to the whole combined query (after any UNION/EXCEPT/INTERSECT), not to an individual SELECT -- so tree.body holds the actual query (a Query or a SetOperation), while tree.order_by/tree.offset/ tree.with_clause live on tree itself. See ast_nodes.SelectExpression's docstring for the full shape.

Package architecture

This package is split into four small, single-purpose modules, plus a sub-package holding the grammar itself:

• grammar/ (lexer.lark, literals.lark, core.lark, adql.lark) The ADQL 2.1 grammar, written for the Lark <https://github.com/lark-parser/lark>_ parsing library (Earley engine), transcribed directly from Annex A of IVOA Recommendation ADQL 2.1. Pure grammar text (no Python), loaded and compiled lazily on first use by parser._get_lark_parser() from grammar/adql.lark (the entry point, which assembles the other three files). Every rule is commented with a cross-reference to the exact Annex A production it implements ([AnnexA #x]). core.lark holds the mutually-recursive heart of the grammar (query structure, predicates, value expressions, functions, geometry) in one file quite deliberately -- see that file's own header comment for the Lark constraints (experimentally confirmed) that make a deeper split unsafe. This is the place to look first when adding support for a new piece of ADQL syntax.

• ast_nodes Defines every AST node as a plain @dataclass: SelectExpression (the root), Query, SetOperation, CTE, TableRef, Join, BinaryOp, Between, InPredicate, Like, IsNull, Exists, Cast/CastType, Coalesce, the geometry nodes Point/Circle/Box/Polygon/Coordinates/ Region/Contains/Intersects/Distance/etc., and more. This module has no dependency on Lark: the AST it defines can be built, inspected, serialized, and tested completely independently of the parser, which keeps the public data model stable even if the parsing engine were ever swapped out.

• parser The bridge between the grammar and the AST: compiles the multi-file Lark grammar (_get_lark_parser(), lazy + cached, resolving grammar/adql.lark's relative %import statements via Lark.open(..., rel_to=...)) and defines ADQLTransformer, a lark.Transformer with one method per grammar rule that turns each Lark parse-tree node into the matching ast_nodes dataclass. Exposes the two public entry points re-exported below: parse_adql (query -> typed AST) and parse_tree (query -> raw Lark tree, mainly useful for debugging the grammar itself). Also emits structured loguru debug logs (grammar-compilation time, Lark-parsing time, AST-building time, AST summary) that the CLI's -v/-vv/-vvv flags surface.

• cli The pyadql command-line entry point (declared in pyproject.toml under [project.scripts]). A thin argparse-based wrapper around parse_adql/parse_tree that reads a query from an argument, a file, or stdin, prints the AST (as a Python repr, JSON, or a raw Lark tree), and configures loguru logging verbosity. Not imported by the two lines below -- it depends on the rest of the package, not the other way around, so importing pyadql as a library never pulls in argparse/CLI-only concerns.

Data flow for a single call to parse_adql(query): query (str) -> Lark parser (grammar/adql.lark + imports, compiled once, cached) -> raw Lark parse tree -> ADQLTransformer (parser.py) -> typed AST (ast_nodes.SelectExpression, wrapping a Query or SetOperation in .body)

parse_adql(query: str)

Parse an ADQL query and return the root of the typed AST (an ast_nodes.SelectExpression; see that class's docstring for the full shape).

Source code in src/pyadql/parser.py

def parse_adql(query: str):
    """Parse an ADQL query and return the root of the typed AST
    (an ast_nodes.SelectExpression; see that class's docstring for the full
    shape)."""
    logger.debug("parse_adql: query of {} characters", len(query))

    t0 = time.perf_counter()
    try:
        tree = _get_lark_parser().parse(query)
    except Exception:
        logger.debug(
            "Failed at the grammar stage (Lark) after {:.1f} ms -- the query "
            "does not follow the expected ADQL/SQL syntax",
            (time.perf_counter() - t0) * 1000,
        )
        raise
    t_parse = time.perf_counter()
    logger.debug(
        "Lark tree obtained in {:.1f} ms ({} direct children under 'start')",
        (t_parse - t0) * 1000,
        len(tree.children),
    )
    logger.opt(lazy=True).trace("Raw Lark tree:\n{}", lambda: tree.pretty())

    try:
        ast = _transformer.transform(tree)
    except Exception:
        logger.debug(
            "Failed at the transformer stage (Lark tree -> typed AST) after "
            "{:.1f} ms -- the grammar matched but building the AST node "
            "failed (check ADQLTransformer in parser.py)",
            (time.perf_counter() - t_parse) * 1000,
        )
        raise
    logger.debug(
        "AST built in {:.1f} ms (total parse+transform: {:.1f} ms): {}",
        (time.perf_counter() - t_parse) * 1000,
        (time.perf_counter() - t0) * 1000,
        _summarize_ast(ast),
    )
    return ast

parse_tree(query: str)

Return the raw Lark parse tree (useful for debugging the grammar).

Source code in src/pyadql/parser.py

def parse_tree(query: str):
    """Return the raw Lark parse tree (useful for debugging the grammar)."""
    logger.debug("parse_tree: query of {} characters", len(query))
    t0 = time.perf_counter()
    tree = _get_lark_parser().parse(query)
    logger.debug("Lark tree obtained in {:.1f} ms", (time.perf_counter() - t0) * 1000)
    return tree

Purpose

This manual serves as a comprehensive resource for:

• Installation and Setup: Guiding you through the installation process and initial configuration.
• Features and Functionality: Explaining the core features and how to use them.
• Best Practices: Offering recommendations for optimal performance and usage.
• Troubleshooting: Providing solutions to common issues and frequently asked questions.

Context

ADQL (Astronomical Data Query Language) is the query language standardized by the IVOA (International Virtual Observatory Alliance) for querying astronomical catalogs through the TAP (Table Access Protocol) protocol. It is a superset of SQL92 enriched with geometry functions (POINT, CIRCLE, BOX, POLYGON, CONTAINS, INTERSECTS, DISTANCE, ...) used to express cross-match or sky-region-search queries. It is the query language spoken by Virtual Observatory data centers such as Gaia, SIMBAD, VizieR, and every other TAP-compliant service.

PyADQL parses an ADQL query into a typed Python abstract syntax tree (AST): plain Python dataclasses (Query, SelectExpression, Join, BinaryOp, Contains, Point, Circle, ...) with no dependency on the parsing engine once the tree is built. This makes PyADQL directly usable to:

• validate an ADQL query before sending it to a TAP service,
• statically analyze a query (referenced columns/tables, pattern detection, etc.),
• transform or rewrite a query (translation to another SQL dialect, injecting security constraints, rewriting subqueries),
• build tooling (linter, editor, autocompletion) around ADQL.

The grammar itself is built with Lark and transcribed directly from Annex A of IVOA Recommendation ADQL 2.1 (2023-12-15), so that every production in the code can be traced back to the standard it implements (see the reference manual and docs/AST.md for details).

This manual is intended for two audiences: developers embedding PyADQL as a library in a larger Python application (a TAP service, a query linter, an IDE plugin, ...), and users running the pyadql command-line tool to inspect or validate a query interactively.