Skip to content

Command-Line Interface

Command-line interface for webdown.

This module provides the command-line interface (CLI) for Webdown, a tool for converting web pages or local HTML files to clean, readable Markdown format. The CLI allows users to customize various aspects of the conversion process, from content selection to formatting options.

For a complete reference, see the CLI Reference documentation.

Basic Usage

The most basic usage is to convert a URL:

webdown -u https://example.com

Or convert a local HTML file:

webdown -f ./page.html

This will convert the content to Markdown, displaying the result to stdout. To save the output to a file:

webdown -u https://example.com -o output.md
webdown -f ./page.html -o output.md

Common Options

The CLI offers various options to customize the conversion:

Source Options

  • -u, --url URL: URL of the web page to convert
  • -f, --file FILE: Path to local HTML file to convert

Input/Output Options

  • -o, --output FILE: Write output to FILE instead of stdout
  • -p, --progress: Show download progress bar (only for URL downloads)

Content Options

  • -t, --toc: Generate a table of contents based on headings
  • -L, --no-links: Strip hyperlinks, converting them to plain text
  • -I, --no-images: Exclude images from the output
  • -s, --css SELECTOR: Extract only content matching the CSS selector (e.g., "main")
  • -c, --compact: Remove excessive blank lines from the output
  • -w, --width N: Set line width for wrapped text (0 for no wrapping)
  • -V, --version: Show version information and exit
  • -h, --help: Show help message and exit

Note: For large web pages (over 10MB), webdown automatically uses streaming mode to optimize memory usage.

Claude XML Options

Options for generating Claude XML format, optimized for use with Claude AI:

  • --claude-xml: Output in Claude XML format instead of Markdown
  • --metadata: Include metadata section in XML (default: True)
  • --no-metadata: Exclude metadata section from XML
  • --no-date: Don't include current date in metadata

Example Scenarios

Web Page Conversion Examples

  1. Basic conversion with a table of contents: 

    webdown -u https://example.com -t -o output.md

  2. Extract only the main content area with compact output and text wrapping: 

    webdown -u https://example.com -s "main" -c -w 80 -o output.md

  3. Create a plain text version (no links or images): 

    webdown -u https://example.com -L -I -o text_only.md

  4. Show download progress for large pages: 

    webdown -u https://example.com -p -o output.md

  5. Extract content from a specific div: 

    webdown -u https://example.com -s "#content" -o output.md

  6. Process a large webpage with progress bar (streaming is automatic for large pages): 

    webdown -u https://example.com -p

  7. Generate output in Claude XML format for use with Claude AI: 

    webdown -u https://example.com -s "main" --claude-xml -o output.xml

  8. Create Claude XML without metadata: 

    webdown -u https://example.com --claude-xml --no-metadata -o output.xml

  9. Complete example with multiple options: 

    webdown -u https://example.com -s "main" -t -c -w 80 -p -o output.md

Local HTML File Conversion Examples

  1. Convert a local HTML file to Markdown: 

    webdown -f ./page.html -o output.md

  2. Convert a local file with table of contents: 

    webdown -f ./page.html -t -o output.md

  3. Extract only main content from a local file: 

    webdown -f ./page.html -s "main" -o output.md

  4. Create plain text from a local file (no links or images): 

    webdown -f ./page.html -L -I -o text_only.md

  5. Convert local file to Claude XML format: 

    webdown -f ./page.html --claude-xml -o output.xml

  6. Complete local file example with multiple options: 

    webdown -f ./page.html -s "main" -t -c -w 80 -o output.md

The entry point is the main() function, which is called when the command webdown is executed.

Functions Reference

Command-line interface for webdown.

This module provides the command-line interface (CLI) for Webdown, a tool for converting web pages or local HTML files to clean, readable Markdown format. The CLI allows users to customize various aspects of the conversion process, from content selection to formatting options.

For a complete reference, see the CLI Reference documentation.

Basic Usage

The most basic usage is to convert a URL:

webdown -u https://example.com

Or convert a local HTML file:

webdown -f ./page.html

This will convert the content to Markdown, displaying the result to stdout. To save the output to a file:

webdown -u https://example.com -o output.md
webdown -f ./page.html -o output.md

Common Options

The CLI offers various options to customize the conversion:

Source Options

  • -u, --url URL: URL of the web page to convert
  • -f, --file FILE: Path to local HTML file to convert

Input/Output Options

  • -o, --output FILE: Write output to FILE instead of stdout
  • -p, --progress: Show download progress bar (only for URL downloads)

Content Options

  • -t, --toc: Generate a table of contents based on headings
  • -L, --no-links: Strip hyperlinks, converting them to plain text
  • -I, --no-images: Exclude images from the output
  • -s, --css SELECTOR: Extract only content matching the CSS selector (e.g., "main")
  • -c, --compact: Remove excessive blank lines from the output
  • -w, --width N: Set line width for wrapped text (0 for no wrapping)
  • -V, --version: Show version information and exit
  • -h, --help: Show help message and exit

Note: For large web pages (over 10MB), webdown automatically uses streaming mode to optimize memory usage.

Claude XML Options

Options for generating Claude XML format, optimized for use with Claude AI:

  • --claude-xml: Output in Claude XML format instead of Markdown
  • --metadata: Include metadata section in XML (default: True)
  • --no-metadata: Exclude metadata section from XML
  • --no-date: Don't include current date in metadata

Example Scenarios

Web Page Conversion Examples

  1. Basic conversion with a table of contents: 

    webdown -u https://example.com -t -o output.md

  2. Extract only the main content area with compact output and text wrapping: 

    webdown -u https://example.com -s "main" -c -w 80 -o output.md

  3. Create a plain text version (no links or images): 

    webdown -u https://example.com -L -I -o text_only.md

  4. Show download progress for large pages: 

    webdown -u https://example.com -p -o output.md

  5. Extract content from a specific div: 

    webdown -u https://example.com -s "#content" -o output.md

  6. Process a large webpage with progress bar (streaming is automatic for large pages): 

    webdown -u https://example.com -p

  7. Generate output in Claude XML format for use with Claude AI: 

    webdown -u https://example.com -s "main" --claude-xml -o output.xml

  8. Create Claude XML without metadata: 

    webdown -u https://example.com --claude-xml --no-metadata -o output.xml

  9. Complete example with multiple options: 

    webdown -u https://example.com -s "main" -t -c -w 80 -p -o output.md

Local HTML File Conversion Examples

  1. Convert a local HTML file to Markdown: 

    webdown -f ./page.html -o output.md

  2. Convert a local file with table of contents: 

    webdown -f ./page.html -t -o output.md

  3. Extract only main content from a local file: 

    webdown -f ./page.html -s "main" -o output.md

  4. Create plain text from a local file (no links or images): 

    webdown -f ./page.html -L -I -o text_only.md

  5. Convert local file to Claude XML format: 

    webdown -f ./page.html --claude-xml -o output.xml

  6. Complete local file example with multiple options: 

    webdown -f ./page.html -s "main" -t -c -w 80 -o output.md

The entry point is the main() function, which is called when the command webdown is executed.

Functions

main(args: Optional[List[str]] = None) -> int

Execute the webdown command-line interface.

This function is the main entry point for the webdown command-line tool. It handles the entire workflow: 1. Parsing command-line arguments 2. Converting the content (URL or file) to Markdown with the specified options 3. Writing the output to a file or stdout 4. Error handling and reporting

Parameters:

Name Type Description Default
args Optional[List[str]]

Command line arguments as a list of strings. If None, defaults to sys.argv[1:] (the command-line arguments passed to the script).

 None

Returns:

Type Description
int

Exit code: 0 for success, 1 for errors

Examples:

>>> main(['-u', 'https://example.com'])  # Convert URL and print to stdout
0
>>> main(['-u', 'https://example.com', '-o', 'output.md'])  # Write URL to file
0
>>> main(['-f', 'page.html'])  # Convert file and print to stdout
0
>>> main(['-f', 'page.html', '-o', 'output.md'])  # Write file to file
0
>>> main(['-u', 'invalid-url'])  # Handle error
1
Source code in webdown/cli.py 
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
def main(args: Optional[List[str]] = None) -> int:
    """Execute the webdown command-line interface.

    This function is the main entry point for the webdown command-line tool.
    It handles the entire workflow:
    1. Parsing command-line arguments
    2. Converting the content (URL or file) to Markdown with the specified options
    3. Writing the output to a file or stdout
    4. Error handling and reporting

    Args:
        args: Command line arguments as a list of strings. If None, defaults to
              sys.argv[1:] (the command-line arguments passed to the script).

    Returns:
        Exit code: 0 for success, 1 for errors

    Examples:
        >>> main(['-u', 'https://example.com'])  # Convert URL and print to stdout
        0
        >>> main(['-u', 'https://example.com', '-o', 'output.md'])  # Write URL to file
        0
        >>> main(['-f', 'page.html'])  # Convert file and print to stdout
        0
        >>> main(['-f', 'page.html', '-o', 'output.md'])  # Write file to file
        0
        >>> main(['-u', 'invalid-url'])  # Handle error
        1
    """
    try:
        # Parse command-line arguments
        parsed_args = parse_args(args)

        # If neither URL nor file provided, show help
        if parsed_args.url is None and parsed_args.file is None:
            # This will print help and exit
            parse_args(["-h"])  # pragma: no cover
            return 0  # pragma: no cover - unreachable after SystemExit

        # Process content and generate output
        content, output_path = _convert_to_selected_format(parsed_args)

        # Write output to file or stdout
        write_output(content, output_path)

        return 0

    except Exception as e:
        # Format and display error message
        formatted_error = format_error_for_cli(e)
        sys.stderr.write(f"{formatted_error}\n")
        return 1

parse_args(args: Optional[List[str]] = None) -> argparse.Namespace

Parse command line arguments.

Parameters:

Name Type Description Default
args Optional[List[str]]

Command line arguments (defaults to sys.argv[1:] if None)

 None

Returns:

Type Description
Namespace

Parsed arguments
Source code in webdown/cli.py 
292
293
294
295
296
297
298
299
300
301
302
def parse_args(args: Optional[List[str]] = None) -> argparse.Namespace:
    """Parse command line arguments.

    Args:
        args: Command line arguments (defaults to sys.argv[1:] if None)

    Returns:
        Parsed arguments
    """
    parser = create_argument_parser()
    return parser.parse_args(args)