Coverage for yuio/config.py: 94%

1# Yuio project, MIT license.

3# https://github.com/taminomara/yuio/

5# You're free to copy this file to your project and edit it for your needs,

6# just keep this copyright line please :3

8"""

9This module provides a base class for configs that can be loaded from

10files, environment variables or command line arguments (via :mod:`yuio.app`).

12Derive your config from the :class:`Config` base class. Inside of its body,

13define config fields using type annotations, just like :mod:`dataclasses`:

15.. code-block:: python

17 class AppConfig(Config):

18 #: trained model to execute

19 model: pathlib.Path

21 #: input data for the model

22 data: pathlib.Path

24 #: enable or disable gpu

25 use_gpu: bool = True

27Then use config's constructors and the :meth:`~Config.update` method

28to load it from various sources::

30 # Load config from a file.

31 config = AppConfig.load_from_json_file("~/.my_app_cfg.json")

33 # Update config with values from env.

34 config.update(AppConfig.load_from_env())

37Config base class

38-----------------

40.. autoclass:: Config

43Advanced field configuration

44----------------------------

46By default, :class:`Config` infers names for env variables and flags,

47appropriate parsers, and other things from field's name, type hint, and comments.

49If you need to override them, you can use :func:`yuio.app.field`

50and :func:`yuio.app.inline` functions (also available from ``yuio.config.field``

51and ``yuio.config.inline``):

53.. code-block:: python

55 class AppConfig(Config):

56 model: pathlib.Path | None = field(

57 default=None,

58 help="trained model to execute",

59 )

62Nesting configs

63---------------

65You can nest configs to achieve modularity:

67.. code-block:: python

69 class ExecutorConfig(Config):

70 #: number of threads to use

71 threads: int

73 #: enable or disable gpu

74 use_gpu: bool = True

77 class AppConfig(Config):

78 #: executor parameters

79 executor: ExecutorConfig

81 #: trained model to execute

82 model: pathlib.Path

84To initialise a nested config, pass either an instance of if

85or a dict with its variables to the config's constructor:

87.. code-block:: python

89 # The following lines are equivalent:

90 config = AppConfig(executor=ExecutorConfig(threads=16))

91 config = AppConfig(executor={"threads": 16})

92 # ...although type checkers will complain about dict =(

95Parsing environment variables

96-----------------------------

98You can load config from environment through :meth:`~Config.load_from_env`.

100Names of environment variables are just capitalized field names.

101Use the :func:`yuio.app.field` function to override them:

102

103.. code-block:: python

104

105 class KillCmdConfig(Config):

106 # Will be loaded from `SIGNAL`.

107 signal: int

108

109 # Will be loaded from `PROCESS_ID`.

110 pid: int = field(env="PROCESS_ID")

111

112In nested configs, environment variable names are prefixed with name

113of a field that contains the nested config:

114

115.. code-block:: python

116

117 class BigConfig(Config):

118 # `kill_cmd.signal` will be loaded from `KILL_CMD_SIGNAL`.

119 kill_cmd: KillCmdConfig

120

121 # `kill_cmd_2.signal` will be loaded from `KILL_SIGNAL`.

122 kill_cmd_2: KillCmdConfig = field(env="KILL")

123

124 # `kill_cmd_3.signal` will be loaded from `SIGNAL`.

125 kill_cmd_3: KillCmdConfig = field(env="")

126

127You can also disable loading a field from an environment altogether:

128

129.. code-block:: python

130

131 class KillCmdConfig(Config):

132 # Will not be loaded from env.

133 pid: int = field(env=yuio.DISABLED)

134

135To prefix all variable names with some string, pass the `prefix` parameter

136to the :meth:`~Config.load_from_env` function:

137

138.. code-block:: python

139

140 # config.kill_cmd.field will be loaded

141 # from `MY_APP_KILL_CMD_SIGNAL`

142 config = BigConfig.load_from_env("MY_APP")

143

144

145Parsing config files

146--------------------

147

148You can load config from structured config files,

149such as `json`, `yaml` or `toml`:

150

151.. skip: next

152

153.. code-block:: python

154

155 class ExecutorConfig(Config):

156 threads: int

157 use_gpu: bool = True

158

159

160 class AppConfig(Config):

161 executor: ExecutorConfig

162 model: pathlib.Path

163

164

165 config = AppConfig.load_from_json_file("~/.my_app_cfg.json")

166

167In this example, contents of the above config would be:

168

169.. code-block:: json

170

171 {

172 "executor": {

173 "threads": 16,

174 "use_gpu": true

175 },

176 "model": "/path/to/model"

177 }

178

179Note that, unlike with environment variables,

180there is no way to inline nested configs.

181

182

183Additional config validation

184----------------------------

185

186If you have invariants that can't be captured with type system,

187you can override :meth:`~Config.validate_config`. This method will be called

188every time you load a config from file, arguments or environment:

189

190.. code-block:: python

191

192 class DocGenConfig(Config):

193 categories: list[str] = ["quickstart", "api_reference"]

194 category_names: dict[str, str] = {"deep_dive": "Deep Dive"}

195

196 def validate_config(self):

197 for category in self.category_names:

198 if category not in self.categories:

199 raise yuio.parse.ParsingError(f"unknown category {category}")

200

201

202Merging configs

203---------------

204

205Configs are specially designed to be merge-able. The basic pattern is to create

206an empty config instance, then :meth:`~Config.update` it with every config source:

207

208.. skip: next

209

210.. code-block:: python

211

212 config = AppConfig()

213 config.update(AppConfig.load_from_json_file("~/.my_app_cfg.json"))

214 config.update(AppConfig.load_from_env())

215 # ...and so on.

216

217The :meth:`~Config.update` function ignores default values, and only overrides

218keys that were actually configured.

219

220If you need a more complex update behavior, you can add a merge function for a field:

221

222.. code-block:: python

223

224 class AppConfig(Config):

225 plugins: list[str] = field(

226 default=[],

227 merge=lambda left, right: [*left, *right],

228 )

229

230Here, whenever we :meth:`~Config.update` ``AppConfig``, ``plugins`` from both instances

231will be concatenated.

232

233.. warning::

234

235 Merge function shouldn't mutate its arguments.

236 It should produce a new value instead.

237

238.. warning::

239

240 Merge function will not be called for default value. It's advisable to keep the

241 default value empty, and add the actual default to the initial empty config:

242

243 .. skip: next

244

245 .. code-block:: python

246

247 config = AppConfig(plugins=["markdown", "rst"])

248 config.update(...)

249

250"""

251

252from __future__ import annotations

253

254import argparse

255import json

256import os

257import pathlib

258import textwrap

259import types

260from dataclasses import dataclass

261

262import yuio

263import yuio.complete

264import yuio.json_schema

265import yuio.parse

266import yuio.string

267from yuio import _typing as _t

268from yuio.util import _find_docs

269

270__all__ = [

271 "Config",

272 "MutuallyExclusiveGroup",

273 "field",

274 "inline",

275 "positional",

276]

277

278T = _t.TypeVar("T")

279Cfg = _t.TypeVar("Cfg", bound="Config")

280

281

282class MutuallyExclusiveGroup:

283 """

284 A sentinel for creating mutually exclusive groups.

285

286 Pass an instance of this class all :func:`~yuio.app.field`\\ s that should

287 be mutually exclusive.

288

289 .. warning::

290

291 Exclusivity checks only work within a single config/command. Passing

292 the same :class:`MutuallyExclusiveGroup` to fields in different configs

293 or commands will not have any effects.

294

295 """

296

297

298@dataclass(frozen=True, slots=True)

299class _FieldSettings:

300 default: _t.Any

301 parser: yuio.parse.Parser[_t.Any] | None = None

302 help: str | yuio.Disabled | None = None

303 env: str | yuio.Disabled | None = None

304 flags: str | list[str] | yuio.Positional | yuio.Disabled | None = None

305 completer: yuio.complete.Completer | None = None

306 metavar: str | None = None

307 required: bool = False

308 merge: _t.Callable[[_t.Any, _t.Any], _t.Any] | None = None

309 group: MutuallyExclusiveGroup | None = None

310 usage: yuio.Group | bool | None = None

311

312 def _update_defaults(

313 self,

314 qualname: str,

315 name: str,

316 ty_with_extras: _t.Any,

317 parsed_help: str | None,

318 allow_positionals: bool,

319 ) -> _Field:

320 ty = ty_with_extras

321 while _t.get_origin(ty) is _t.Annotated:

322 ty = _t.get_args(ty)[0]

323 is_subconfig = isinstance(ty, type) and issubclass(ty, Config)

324

325 help: str | yuio.Disabled

326 if self.help is not None:

327 help = self.help

328 elif parsed_help is not None:

329 help = parsed_help

330 elif is_subconfig and ty.__doc__:

331 help = ty.__doc__

332 else:

333 help = ""

334 if help == argparse.SUPPRESS:

335 help = yuio.DISABLED

336

337 env: str | yuio.Disabled

338 if self.env is not None:

339 env = self.env

340 else:

341 env = name.upper()

342 if env == "" and not is_subconfig:

343 raise TypeError(f"{qualname} got an empty env variable name")

344

345 flags: list[str] | yuio.Positional | yuio.Disabled

346 if self.flags is yuio.DISABLED or self.flags is yuio.POSITIONAL:

347 flags = self.flags

348 if not allow_positionals and flags is yuio.POSITIONAL:

349 raise TypeError(

350 f"{qualname}: positional arguments are not allowed in configs"

351 )

352 elif self.flags is None:

353 flags = ["--" + name.replace("_", "-")]

354 elif isinstance(self.flags, str):

355 flags = [self.flags]

356 else:

357 if not self.flags:

358 raise TypeError(f"{qualname} should have at least one flag")

359 flags = self.flags

360 if flags is not yuio.DISABLED and flags is not yuio.POSITIONAL:

361 for flag in flags:

362 if flag and not flag.startswith("-"):

363 raise TypeError(f"{qualname}: flag should start with a dash")

364 if not flag and not is_subconfig:

365 raise TypeError(f"{qualname} got an empty flag")

366

367 default = self.default

368

369 parser = self.parser

370

371 required = self.required

372

373 if is_subconfig:

374 if default is not yuio.MISSING:

375 raise TypeError(

376 f"error in {qualname}: nested configs can't have defaults"

377 )

378

379 if parser is not None:

380 raise TypeError(

381 f"error in {qualname}: nested configs can't have parsers"

382 )

383

384 if flags is not yuio.DISABLED:

385 if flags is yuio.POSITIONAL:

386 raise TypeError(

387 f"error in {qualname}: nested configs can't be positional"

388 )

389 if len(flags) > 1:

390 raise TypeError(

391 f"error in {qualname}: nested configs can't have multiple flags"

392 )

393 if flags[0] and not flags[0].startswith("--"):

394 raise TypeError(

395 f"error in {qualname}: nested configs can't have a short flag"

396 )

397 elif parser is None:

398 try:

399 parser = yuio.parse.from_type_hint(ty_with_extras)

400 except TypeError as e:

401 raise TypeError(

402 f"can't derive parser for {qualname}:\n"

403 + textwrap.indent(str(e), " ")

404 ) from None

405

406 if parser is not None:

407 origin = _t.get_origin(ty)

408 args = _t.get_args(ty)

409

410 is_optional = (

411 default is None or _t.is_union(origin) and types.NoneType in args

412 )

413

414 if is_optional and not yuio.parse._is_optional_parser(parser):

415 parser = yuio.parse.Optional(parser)

416

417 if (

418 flags is yuio.POSITIONAL

419 and default is not yuio.MISSING

420 and parser.supports_parse_many()

421 ):

422 raise TypeError(

423 f"{qualname}: positional multi-value arguments can't have defaults"

424 )

425

426 completer = self.completer

427 metavar = self.metavar

428 if parser is not None and (completer is not None or metavar is not None):

429 parser = yuio.parse.WithMeta(parser, desc=metavar, completer=completer)

430

431 merge = self.merge

432

433 if is_subconfig and merge is not None:

434 raise TypeError(f"error in {qualname}: nested configs can't have merge")

435

436 group = self.group

437

438 if is_subconfig and group is not None:

439 raise TypeError(

440 f"error in {qualname}: nested configs can't be a part "

441 "of a mutually exclusive group"

442 )

443

444 usage = self.usage

445

446 return _Field(

447 default,

448 parser,

449 help,

450 env,

451 flags,

452 is_subconfig,

453 ty,

454 required,

455 merge,

456 group,

457 usage,

458 )

459

460

461@dataclass(frozen=True, slots=True)

462class _Field:

463 default: _t.Any

464 parser: yuio.parse.Parser[_t.Any] | None

465 help: str | yuio.Disabled

466 env: str | yuio.Disabled

467 flags: list[str] | yuio.Positional | yuio.Disabled

468 is_subconfig: bool

469 ty: type

470 required: bool

471 merge: _t.Callable[[_t.Any, _t.Any], _t.Any] | None

472 group: MutuallyExclusiveGroup | None

473 usage: yuio.Group | bool | None = None

474

475

476@_t.overload

477def field(

478 *,

479 completer: yuio.complete.Completer | None = None,

480 metavar: str | None = None,

481 help: str | yuio.Disabled | None = None,

482 env: str | yuio.Disabled | None = None,

483 flags: str | list[str] | yuio.Positional | yuio.Disabled | None = None,

484 group: MutuallyExclusiveGroup | None = None,

485 usage: yuio.Group | bool | None = None,

486) -> _t.Any: ...

487@_t.overload

488def field(

489 *,

490 default: None,

491 parser: yuio.parse.Parser[T] | None = None,

492 help: str | yuio.Disabled | None = None,

493 env: str | yuio.Disabled | None = None,

494 flags: str | list[str] | yuio.Positional | yuio.Disabled | None = None,

495 completer: yuio.complete.Completer | None = None,

496 metavar: str | None = None,

497 merge: _t.Callable[[T, T], T] | None = None,

498 group: MutuallyExclusiveGroup | None = None,

499 usage: yuio.Group | bool | None = None,

500) -> T | None: ...

501@_t.overload

502def field(

503 *,

504 default: T | yuio.Missing = yuio.MISSING,

505 parser: yuio.parse.Parser[T] | None = None,

506 help: str | yuio.Disabled | None = None,

507 env: str | yuio.Disabled | None = None,

508 flags: str | list[str] | yuio.Positional | yuio.Disabled | None = None,

509 completer: yuio.complete.Completer | None = None,

510 metavar: str | None = None,

511 merge: _t.Callable[[T, T], T] | None = None,

512 group: MutuallyExclusiveGroup | None = None,

513 usage: yuio.Group | bool | None = None,

514) -> T: ...

515def field(

516 *,

517 default: _t.Any = yuio.MISSING,

518 parser: yuio.parse.Parser[_t.Any] | None = None,

519 help: str | yuio.Disabled | None = None,

520 env: str | yuio.Disabled | None = None,

521 flags: str | list[str] | yuio.Positional | yuio.Disabled | None = None,

522 completer: yuio.complete.Completer | None = None,

523 metavar: str | None = None,

524 merge: _t.Callable[[_t.Any, _t.Any], _t.Any] | None = None,

525 group: MutuallyExclusiveGroup | None = None,

526 usage: yuio.Group | bool | None = None,

527) -> _t.Any:

528 """

529 Field descriptor, used for additional configuration of CLI arguments

530 and config fields.

531

532 :param default:

533 default value for the field or CLI argument.

534 :param parser:

535 parser that will be used to parse config values and CLI arguments.

536 :param help:

537 help message that will be used in CLI argument description.

538

539 Pass :data:`~yuio.DISABLED` to remove this field from CLI help.

540

541 Help messages are formatted using Markdown (see :mod:`yuio.md`).

542

543 By default, first paragraph of the field's documentation is used.

544 The documentation is processed using markdown parser; additionally, RST roles

545 are processed, trailing dot is removed, and capitalization is normalized

546 to match style of default help messages.

547 :param env:

548 in configs, specifies name of environment variable that will be used

549 if loading config from environment variable.

550

551 Pass :data:`~yuio.DISABLED` to disable loading this field form environment variable.

552

553 Pass an empty string to disable prefixing nested config variables.

554 :param flags:

555 list of names (or a single name) of CLI flags that will be used for this field.

556

557 In configs, pass :data:`~yuio.DISABLED` to disable loading this field form CLI arguments.

558

559 In apps, pass :data:`~yuio.POSITIONAL` to make this argument positional.

560

561 Pass an empty string to disable prefixing nested config flags.

562 :param completer:

563 completer that will be used for autocompletion in CLI. Using this option

564 is equivalent to overriding ``completer`` with :class:`yuio.parse.WithMeta`.

565 :param metavar:

566 value description that will be used for CLI help messages. Using this option

567 is equivalent to overriding ``desc`` with :class:`yuio.parse.WithMeta`.

568 :param merge:

569 defines how values of this field are merged when configs are updated.

570 :param group:

571 defines mutually exclusive group for this field. Create an instance

572 of :class:`yuio.app.MutuallyExclusiveGroup` and pass it to all fields that

573 should be mutually exclusive.

574 :param usage:

575 controls how this field renders in CLI usage section. Passing :data:`False`

576 removes this field from usage, and passing :class:`yuio.GROUP` replaces all

577 omitted fields with a single string ``"<options>"``.

578 :returns:

579 a magic object that will be replaced with field's default value once a new

580 config class is created.

581 :example:

582 In apps:

583

584 .. invisible-code-block: python

585

586 import yuio.app

587

588 .. code-block:: python

589

590 @yuio.app.app

591 def main(

592 # Will be loaded from `--input`.

593 input: pathlib.Path | None = None,

594

595 # Will be loaded from `-o` or `--output`.

596 output: pathlib.Path | None = field(default=None, flags=['-p', '--pid'])

597 ):

598 ...

599 :example:

600 In configs:

601

602 .. code-block:: python

603

604 class AppConfig(Config):

605 model: pathlib.Path | None = field(

606 default=None,

607 help="trained model to execute",

608 )

609

610 """

611

612 return _FieldSettings(

613 default=default,

614 completer=completer,

615 metavar=metavar,

616 parser=parser,

617 help=help,

618 env=env,

619 flags=flags,

620 merge=merge,

621 group=group,

622 usage=usage,

623 )

624

625

626def inline(

627 help: str | yuio.Disabled | None = None,

628 usage: yuio.Group | bool | None = None,

629) -> _t.Any:

630 """

631 A shortcut for inlining nested configs.

632

633 Equivalent to calling :func:`~yuio.app.field` with ``env`` and ``flags``

634 set to an empty string.

635

636 """

637

638 return field(help=help, env="", flags="", usage=usage)

639

640

641@_t.overload

642def positional(

643 *,

644 help: str | yuio.Disabled | None = None,

645 env: str | yuio.Disabled | None = None,

646 completer: yuio.complete.Completer | None = None,

647 metavar: str | None = None,

648 usage: yuio.Group | bool | None = None,

649) -> _t.Any: ...

650@_t.overload

651def positional(

652 *,

653 default: None,

654 parser: yuio.parse.Parser[T] | None = None,

655 help: str | yuio.Disabled | None = None,

656 env: str | yuio.Disabled | None = None,

657 completer: yuio.complete.Completer | None = None,

658 metavar: str | None = None,

659 usage: yuio.Group | bool | None = None,

660) -> T | None: ...

661@_t.overload

662def positional(

663 *,

664 default: T | yuio.Missing = yuio.MISSING,

665 parser: yuio.parse.Parser[T] | None = None,

666 help: str | yuio.Disabled | None = None,

667 env: str | yuio.Disabled | None = None,

668 completer: yuio.complete.Completer | None = None,

669 metavar: str | None = None,

670 usage: yuio.Group | bool | None = None,

671) -> T: ...

672def positional(

673 *,

674 default: _t.Any = yuio.MISSING,

675 parser: yuio.parse.Parser[_t.Any] | None = None,

676 help: str | yuio.Disabled | None = None,

677 env: str | yuio.Disabled | None = None,

678 completer: yuio.complete.Completer | None = None,

679 metavar: str | None = None,

680 usage: yuio.Group | bool | None = None,

681) -> _t.Any:

682 """

683 A shortcut for adding a positional argument.

684

685 Equivalent to calling :func:`field` with ``flags`` set to :data:`~yuio.POSITIONAL`.

686

687 """

688

689 return field(

690 default=default,

691 parser=parser,

692 help=help,

693 env=env,

694 flags=yuio.POSITIONAL,

695 completer=completer,

696 metavar=metavar,

697 usage=usage,

698 )

699

700

701def _action(

702 field: _Field,

703 usage: yuio.Group | bool,

704 parse_many: bool = False,

705 const: _t.Any = yuio.MISSING,

706):

707 class Action(argparse.Action):

708 @staticmethod

709 def get_parser():

710 return field.parser

711

712 @staticmethod

713 def get_merge():

714 return field.merge

715

716 @staticmethod

717 def get_usage():

718 return usage

719

720 def __call__(self, _, namespace, values, option_string=None):

721 try:

722 if const is not yuio.MISSING:

723 assert field.parser

724 assert values == []

725 parsed = field.parser.parse_config(const)

726 elif parse_many:

727 if values is yuio.MISSING:

728 values = []

729 assert values is not None

730 assert not isinstance(values, str)

731 assert field.parser

732 parsed = field.parser.parse_many(values)

733 else:

734 if values is yuio.MISSING:

735 return

736 assert isinstance(values, str)

737 assert field.parser

738 parsed = field.parser.parse(values)

739 except argparse.ArgumentTypeError as e:

740 raise argparse.ArgumentError(self, str(e))

741 # Note: merge will be executed in `namespace.__setattr__`,

742 # see `yuio.app._Namespace`.

743 setattr(namespace, self.dest, parsed)

744

745 return Action

746

747

748@_t.dataclass_transform(

749 eq_default=False,

750 order_default=False,

751 kw_only_default=True,

752 frozen_default=False,

753 field_specifiers=(field, inline, positional),

754)

755class Config:

756 """

757 Base class for configs.

758

759 Pass keyword args to set fields, or pass another config to copy it::

760

761 Config(config1, config2, ..., field1=value1, ...)

762

763 Upon creation, all fields that aren't explicitly initialized

764 and don't have defaults are considered missing.

765 Accessing them will raise :class:`AttributeError`.

766

767 .. automethod:: update

768

769 .. automethod:: load_from_env

770

771 .. automethod:: load_from_json_file

772

773 .. automethod:: load_from_yaml_file

774

775 .. automethod:: load_from_toml_file

776

777 .. automethod:: load_from_parsed_file

778

779 .. automethod:: to_json_schema

780

781 .. automethod:: to_json_value

782

783 .. automethod:: validate_config

784

785 """

786

787 @classmethod

788 def __get_fields(cls) -> dict[str, _Field]:

789 if cls.__fields is not None:

790 return cls.__fields

791

792 if cls.__allow_positionals:

793 docs = {}

794 else:

795 try:

796 docs = _find_docs(cls)

797 except Exception:

798 yuio._logger.warning(

799 "unable to get documentation for class %s.%s",

800 cls.__module__,

801 cls.__qualname__,

802 )

803 docs = {}

804

805 fields = {}

806

807 for base in reversed(cls.__mro__):

808 if base is not cls and hasattr(base, "_Config__get_fields"):

809 fields.update(getattr(base, "_Config__get_fields")())

810

811 try:

812 types = _t.get_type_hints(cls, include_extras=True)

813 except NameError as e:

814 if "<locals>" in cls.__qualname__:

815 raise NameError(

816 f"{e}. "

817 f"Note: forward references do not work inside functions "

818 f"(see https://github.com/python/typing/issues/797)"

819 ) from None

820 raise

821

822 for name, field in cls.__gathered_fields.items():

823 if not isinstance(field, _FieldSettings):

824 field = _FieldSettings(default=field)

825

826 fields[name] = field._update_defaults(

827 f"{cls.__qualname__}.{name}",

828 name,

829 types[name],

830 docs.get(name),

831 cls.__allow_positionals,

832 )

833 cls.__fields = fields

834

835 return fields

836

837 def __init_subclass__(cls, _allow_positionals=None, **kwargs):

838 super().__init_subclass__(**kwargs)

839

840 if _allow_positionals is not None:

841 cls.__allow_positionals: bool = _allow_positionals

842 cls.__fields: dict[str, _Field] | None = None

843

844 cls.__gathered_fields: dict[str, _FieldSettings | _t.Any] = {}

845 for name in cls.__annotations__:

846 if not name.startswith("_"):

847 cls.__gathered_fields[name] = cls.__dict__.get(name, yuio.MISSING)

848 for name, value in cls.__dict__.items():

849 if isinstance(value, _FieldSettings) and name not in cls.__gathered_fields:

850 qualname = f"{cls.__qualname__}.{name}"

851 raise TypeError(

852 f"error in {qualname}: field without annotations is not allowed"

853 )

854 for name, value in cls.__gathered_fields.items():

855 if isinstance(value, _FieldSettings):

856 value = value.default

857 setattr(cls, name, value)

858

859 def __init__(self, *args: _t.Self | dict[str, _t.Any], **kwargs):

860 for name, field in self.__get_fields().items():

861 if field.is_subconfig:

862 setattr(self, name, field.ty())

863

864 for arg in args:

865 self.update(arg)

866

867 self.update(kwargs)

868

869 def update(self, other: _t.Self | dict[str, _t.Any], /):

870 """

871 Update fields in this config with fields from another config.

872

873 This function is similar to :meth:`dict.update`.

874

875 Nested configs are updated recursively.

876

877 :param other:

878 data for update.

879

880 """

881

882 if not other:

883 return

884

885 if isinstance(other, Config):

886 if (

887 self.__class__ not in other.__class__.__mro__

888 and other.__class__ not in self.__class__.__mro__

889 ):

890 raise TypeError("updating from an incompatible config")

891 ns = other.__dict__

892 elif isinstance(other, dict):

893 ns = other

894 for name in ns:

895 if name not in self.__get_fields():

896 raise TypeError(f"unknown field: {name}")

897 else:

898 raise TypeError("expected a dict or a config class")

899

900 for name, field in self.__get_fields().items():

901 if name in ns:

902 if field.is_subconfig:

903 getattr(self, name).update(ns[name])

904 elif ns[name] is not yuio.MISSING:

905 if field.merge is not None and name in self.__dict__:

906 setattr(self, name, field.merge(getattr(self, name), ns[name]))

907 else:

908 setattr(self, name, ns[name])

909

910 @classmethod

911 def load_from_env(cls, prefix: str = "") -> _t.Self:

912 """

913 Load config from environment variables.

914

915 :param prefix:

916 if given, names of all environment variables will be prefixed with

917 this string and an underscore.

918 :returns:

919 a parsed config.

920 :raises:

921 :class:`~yuio.parse.ParsingError`.

922

923 """

924

925 try:

926 result = cls.__load_from_env(prefix)

927 result.validate_config()

928 return result

929 except yuio.parse.ParsingError as e:

930 raise yuio.parse.ParsingError(

931 "Failed to load config from environment variables:\n%s",

932 yuio.string.Indent(e),

933 ) from None

934

935 @classmethod

936 def __load_from_env(cls, prefix: str = "") -> _t.Self:

937 fields = {}

938

939 for name, field in cls.__get_fields().items():

940 if field.env is yuio.DISABLED:

941 continue

942

943 if prefix and field.env:

944 env = f"{prefix}_{field.env}"

945 else:

946 env = f"{prefix}{field.env}"

947

948 if field.is_subconfig:

949 fields[name] = field.ty.load_from_env(prefix=env)

950 elif env in os.environ:

951 assert field.parser is not None

952 fields[name] = field.parser.parse(os.environ[env])

953

954 return cls(**fields)

955

956 @classmethod

957 def _load_from_namespace(

958 cls,

959 namespace: argparse.Namespace,

960 /,

961 *,

962 ns_prefix: str = "",

963 ) -> _t.Self:

964 result = cls.__load_from_namespace(namespace, ns_prefix + ":")

965 result.validate_config()

966 return result

967

968 @classmethod

969 def __load_from_namespace(

970 cls, namespace: argparse.Namespace, prefix: str

971 ) -> _t.Self:

972 fields = {}

973

974 for name, field in cls.__get_fields().items():

975 if field.flags is yuio.DISABLED:

976 continue

977

978 dest = prefix + name

979

980 if field.is_subconfig:

981 fields[name] = field.ty.__load_from_namespace(namespace, dest + ".")

982 elif hasattr(namespace, dest):

983 fields[name] = getattr(namespace, dest)

984

985 return cls(**fields)

986

987 @classmethod

988 def _setup_arg_parser(

989 cls,

990 parser: argparse.ArgumentParser,

991 /,

992 *,

993 group: argparse.ArgumentParser | None = None,

994 ns_prefix: str = "",

995 ):

996 group = group or parser

997 cls.__setup_arg_parser(group, parser, "", ns_prefix + ":", False, 0, True)

998

999 @classmethod

1000 def __setup_arg_parser(

1001 cls,

1002 group: argparse.ArgumentParser,

1003 parser: argparse.ArgumentParser,

1004 prefix: str,

1005 dest_prefix: str,

1006 suppress_help: bool,

1007 depth: int,

1008 usage: yuio.Group | bool,

1009 ):

1010 if prefix:

1011 prefix += "-"

1012

1013 mutex_groups = {}

1014

1015 for name, field in cls.__get_fields().items():

1016 if field.flags is yuio.DISABLED:

1017 continue

1018

1019 dest = dest_prefix + name

1020

1021 if suppress_help or field.help is yuio.DISABLED:

1022 help = argparse.SUPPRESS

1023 current_suppress_help = True

1024 current_usage = field.usage if field.usage is not None else False

1025 else:

1026 help = field.help

1027 current_suppress_help = False

1028 current_usage = field.usage if field.usage is not None else usage

1029

1030 flags: list[str] | yuio.Positional

1031 if prefix and field.flags is not yuio.POSITIONAL:

1032 flags = [prefix + flag.lstrip("-") for flag in field.flags]

1033 else:

1034 flags = field.flags

1035

1036 if field.is_subconfig:

1037 assert flags is not yuio.POSITIONAL

1038 if current_suppress_help:

1039 subgroup = group

1040 else:

1041 lines = help.split("\n\n", 1)

1042 title = lines[0].replace("\n", " ").rstrip(".") or name

1043 desc = textwrap.dedent(lines[1]) if len(lines) > 1 else None

1044 subgroup = parser.add_argument_group(title, desc)

1045 field.ty.__setup_arg_parser(

1046 subgroup,

1047 parser,

1048 flags[0],

1049 dest + ".",

1050 current_suppress_help,

1051 depth + 1,

1052 current_usage,

1053 )

1054 continue

1055 else:

1056 assert field.parser is not None

1057

1058 parse_many = field.parser.supports_parse_many()

1059

1060 if flags is yuio.POSITIONAL:

1061 metavar = f"<{name.replace('_', '-')}>"

1062 elif parse_many:

1063 metavar = field.parser.describe_many()

1064 else:

1065 metavar = field.parser.describe_or_def()

1066

1067 nargs = field.parser.get_nargs()

1068 if (

1069 flags is yuio.POSITIONAL

1070 and field.default is not yuio.MISSING

1071 and nargs is None

1072 ):

1073 nargs = "?"

1074 nargs_kw: _t.Any = {"nargs": nargs} if nargs is not None else {}

1075

1076 if field.group is not None:

1077 if field.group not in mutex_groups:

1078 mutex_groups[field.group] = group.add_mutually_exclusive_group()

1079 field_group = mutex_groups[field.group]

1080 else:

1081 field_group = None

1082

1083 if flags is yuio.POSITIONAL:

1084 (field_group or group).add_argument(

1085 dest,

1086 default=yuio.MISSING,

1087 help=help,

1088 metavar=metavar,

1089 action=_action(field, current_usage, parse_many),

1090 **nargs_kw,

1091 )

1092 elif yuio.parse._is_bool_parser(field.parser):

1093 mutex_group = field_group or group.add_mutually_exclusive_group(

1094 required=field.required

1095 )

1096

1097 if depth > 0 or not isinstance(field.default, bool):

1098 pos_help = help

1099 neg_help = None

1100 elif field.default:

1101 pos_help = argparse.SUPPRESS

1102 neg_help = help

1103 else:

1104 pos_help = help

1105 neg_help = argparse.SUPPRESS

1106

1107 mutex_group.add_argument(

1108 *flags,

1109 default=yuio.MISSING,

1110 help=pos_help,

1111 dest=dest,

1112 action=_action(field, current_usage, const=True),

1113 nargs=0,

1114 )

1115

1116 assert field.flags is not yuio.POSITIONAL

1117 for flag in field.flags:

1118 if flag.startswith("--"):

1119 flag_neg = (prefix or "--") + "no-" + flag[2:]

1120 if current_suppress_help or neg_help == argparse.SUPPRESS:

1121 help = argparse.SUPPRESS

1122 elif neg_help:

1123 help = neg_help

1124 else:

1125 help = f"disable <c hl/flag:sh-usage>{(prefix or '--') + flag[2:]}</c>"

1126 mutex_group.add_argument(

1127 flag_neg,

1128 default=yuio.MISSING,

1129 help=help,

1130 dest=dest,

1131 action=_action(field, current_usage, const=False),

1132 nargs=0,

1133 )

1134 break

1135 else:

1136 (field_group or group).add_argument(

1137 *flags,

1138 default=yuio.MISSING,

1139 help=help,

1140 metavar=metavar,

1141 required=field.required,

1142 dest=dest,

1143 action=_action(field, current_usage, parse_many),

1144 **nargs_kw,

1145 )

1146

1147 @classmethod

1148 def load_from_json_file(

1149 cls,

1150 path: str | pathlib.Path,

1151 /,

1152 *,

1153 ignore_unknown_fields: bool = False,

1154 ignore_missing_file: bool = False,

1155 ) -> _t.Self:

1156 """

1157 Load config from a ``.json`` file.

1158

1159 :param path:

1160 path of the config file.

1161 :param ignore_unknown_fields:

1162 if :data:`True`, this method will ignore fields that aren't listed

1163 in config class.

1164 :param ignore_missing_file:

1165 if :data:`True`, silently ignore a missing file error. This is useful

1166 when loading a config from a home directory.

1167 :returns:

1168 a parsed config.

1169 :raises:

1170 :class:`~yuio.parse.ParsingError` if config parsing has failed

1171 or if config file doesn't exist.

1172

1173 """

1174

1175 return cls.__load_from_file(

1176 path, json.loads, ignore_unknown_fields, ignore_missing_file

1177 )

1178

1179 @classmethod

1180 def load_from_yaml_file(

1181 cls,

1182 path: str | pathlib.Path,

1183 /,

1184 *,

1185 ignore_unknown_fields: bool = False,

1186 ignore_missing_file: bool = False,

1187 ) -> _t.Self:

1188 """

1189 Load config from a ``.yaml`` file.

1190

1191 This requires `PyYaml <https://pypi.org/project/PyYAML/>`__ package

1192 to be installed.

1193

1194 :param path:

1195 path of the config file.

1196 :param ignore_unknown_fields:

1197 if :data:`True`, this method will ignore fields that aren't listed

1198 in config class.

1199 :param ignore_missing_file:

1200 if :data:`True`, silently ignore a missing file error. This is useful

1201 when loading a config from a home directory.

1202 :returns:

1203 a parsed config.

1204 :raises:

1205 :class:`~yuio.parse.ParsingError` if config parsing has failed

1206 or if config file doesn't exist. Can raise :class:`ImportError`

1207 if ``PyYaml`` is not available.

1208

1209 """

1210

1211 try:

1212 import yaml

1213 except ImportError:

1214 raise ImportError("PyYaml is not available")

1215

1216 return cls.__load_from_file(

1217 path, yaml.safe_load, ignore_unknown_fields, ignore_missing_file

1218 )

1219

1220 @classmethod

1221 def load_from_toml_file(

1222 cls,

1223 path: str | pathlib.Path,

1224 /,

1225 *,

1226 ignore_unknown_fields: bool = False,

1227 ignore_missing_file: bool = False,

1228 ) -> _t.Self:

1229 """

1230 Load config from a ``.toml`` file.

1231

1232 This requires

1233 `tomllib <https://docs.python.org/3/library/tomllib.html>`_ or

1234 `toml <https://pypi.org/project/toml/>`_ package

1235 to be installed.

1236

1237 :param path:

1238 path of the config file.

1239 :param ignore_unknown_fields:

1240 if :data:`True`, this method will ignore fields that aren't listed

1241 in config class.

1242 :param ignore_missing_file:

1243 if :data:`True`, silently ignore a missing file error. This is useful

1244 when loading a config from a home directory.

1245 :returns:

1246 a parsed config.

1247 :raises:

1248 :class:`~yuio.parse.ParsingError` if config parsing has failed

1249 or if config file doesn't exist. Can raise :class:`ImportError`

1250 if ``toml`` is not available.

1251

1252 """

1253

1254 try:

1255 import toml

1256 except ImportError:

1257 try:

1258 import tomllib as toml # type: ignore

1259 except ImportError:

1260 raise ImportError("toml is not available")

1261

1262 return cls.__load_from_file(

1263 path, toml.loads, ignore_unknown_fields, ignore_missing_file

1264 )

1265

1266 @classmethod

1267 def __load_from_file(

1268 cls,

1269 path: str | pathlib.Path,

1270 file_parser: _t.Callable[[str], _t.Any],

1271 ignore_unknown_fields: bool = False,

1272 ignore_missing_file: bool = False,

1273 ) -> _t.Self:

1274 path = pathlib.Path(path)

1275

1276 if ignore_missing_file and (not path.exists() or not path.is_file()):

1277 return cls()

1278

1279 try:

1280 loaded = file_parser(path.read_text())

1281 except Exception as e:

1282 raise yuio.parse.ParsingError(

1283 "Invalid config <c path>%s</c>:\n%s",

1284 path,

1285 yuio.string.Indent(e),

1286 ) from None

1287

1288 return cls.load_from_parsed_file(

1289 loaded, ignore_unknown_fields=ignore_unknown_fields, path=path

1290 )

1291

1292 @classmethod

1293 def load_from_parsed_file(

1294 cls,

1295 parsed: dict[str, object],

1296 /,

1297 *,

1298 ignore_unknown_fields: bool = False,

1299 path: str | pathlib.Path | None = None,

1300 ) -> _t.Self:

1301 """

1302 Load config from parsed config file.

1303

1304 This method takes a dict with arbitrary values that you'd get from

1305 parsing type-rich configs such as ``yaml`` or ``json``.

1306

1307 For example::

1308

1309 with open("conf.yaml") as file:

1310 config = Config.load_from_parsed_file(yaml.load(file))

1311

1312 :param parsed:

1313 data from parsed file.

1314 :param ignore_unknown_fields:

1315 if :data:`True`, this method will ignore fields that aren't listed

1316 in config class.

1317 :param path:

1318 path of the original file, used for error reporting.

1319 :returns:

1320 a parsed config.

1321 :raises:

1322 :class:`~yuio.parse.ParsingError`.

1323

1324 """

1325

1326 try:

1327 result = cls.__load_from_parsed_file(parsed, ignore_unknown_fields, "")

1328 result.validate_config()

1329 return result

1330 except yuio.parse.ParsingError as e:

1331 if path is None:

1332 raise

1333 else:

1334 raise yuio.parse.ParsingError(

1335 "Invalid config <c path>%s</c>:\n%s",

1336 path,

1337 yuio.string.Indent(e),

1338 ) from None

1339

1340 @classmethod

1341 def __load_from_parsed_file(

1342 cls,

1343 parsed: dict[str, object],

1344 ignore_unknown_fields: bool = False,

1345 field_prefix: str = "",

1346 ) -> _t.Self:

1347 if not isinstance(parsed, dict):

1348 raise TypeError("config should be a dict")

1349

1350 fields = {}

1351

1352 if not ignore_unknown_fields:

1353 for name in parsed:

1354 if name not in cls.__get_fields() and name != "$schema":

1355 raise yuio.parse.ParsingError(

1356 "Unknown field `%s`", f"{field_prefix}{name}"

1357 )

1358

1359 for name, field in cls.__get_fields().items():

1360 if name in parsed:

1361 if field.is_subconfig:

1362 fields[name] = field.ty.__load_from_parsed_file(

1363 parsed[name], ignore_unknown_fields, field_prefix=name + "."

1364 )

1365 else:

1366 assert field.parser is not None

1367 try:

1368 value = field.parser.parse_config(parsed[name])

1369 except yuio.parse.ParsingError as e:

1370 raise yuio.parse.ParsingError(

1371 "Can't parse field `%s%s`:\n%s",

1372 field_prefix,

1373 name,

1374 yuio.string.Indent(e),

1375 ) from None

1376 fields[name] = value

1377

1378 return cls(**fields)

1379

1380 def __getattribute(self, item):

1381 value = super().__getattribute__(item)

1382 if value is yuio.MISSING:

1383 raise AttributeError(f"{item} is not configured")

1384 else:

1385 return value

1386

1387 # A dirty hack to hide `__getattribute__` from type checkers.

1388 locals()["__getattribute__"] = __getattribute

1389

1390 def __repr__(self):

1391 field_reprs = ", ".join(

1392 f"{name}={getattr(self, name, yuio.MISSING)!r}"

1393 for name in self.__get_fields()

1394 )

1395 return f"{self.__class__.__name__}({field_reprs})"

1396

1397 def __rich_repr__(self):

1398 for name in self.__get_fields():

1399 yield name, getattr(self, name, yuio.MISSING)

1400

1401 def validate_config(self):

1402 """

1403 Perform config validation.

1404

1405 This function is called every time a config is loaded from CLI arguments,

1406 file, or environment variables. It should check that config is correct,

1407 and raise :class:`yuio.parse.ParsingError` if it's not.

1408

1409 :raises:

1410 :class:`~yuio.parse.ParsingError`.

1411

1412 """

1413

1414 @classmethod

1415 def to_json_schema(

1416 cls, ctx: yuio.json_schema.JsonSchemaContext

1417 ) -> yuio.json_schema.JsonSchemaType:

1418 """

1419 Create a JSON schema object based on this config.

1420

1421 The purpose of this method is to make schemas for use in IDEs, i.e. to provide

1422 autocompletion or simple error checking. The returned schema is not guaranteed

1423 to reflect all constraints added to the parser.

1424

1425 :param ctx:

1426 context for building a schema.

1427 :returns:

1428 a JSON schema that describes structure of this config.

1429

1430 """

1431

1432 return ctx.add_type(cls, _t.type_repr(cls), lambda: cls.__to_json_schema(ctx))

1433

1434 def to_json_value(

1435 self, *, include_defaults: bool = True

1436 ) -> yuio.json_schema.JsonValue:

1437 """

1438 Convert this config to a representation suitable for JSON serialization.

1439

1440 :param include_defaults:

1441 if :data:`False`, default values will be skipped.

1442 :returns:

1443 a config converted to JSON-serializable representation.

1444 :raises:

1445 :class:`TypeError` if any of the config fields contain values that can't

1446 be converted to JSON by their respective parsers.

1447

1448 """

1449

1450 data = {}

1451 for name, field in self.__get_fields().items():

1452 if not include_defaults and name not in self.__dict__:

1453 continue

1454 if field.is_subconfig:

1455 value = getattr(self, name).to_json_value(

1456 include_defaults=include_defaults

1457 )

1458 if value:

1459 data[name] = value

1460 else:

1461 assert field.parser

1462 try:

1463 value = getattr(self, name)

1464 except AttributeError:

1465 pass

1466 else:

1467 data[name] = field.parser.to_json_value(value)

1468 return data

1469

1470 @classmethod

1471 def __to_json_schema(

1472 cls, ctx: yuio.json_schema.JsonSchemaContext

1473 ) -> yuio.json_schema.JsonSchemaType:

1474 properties: dict[str, yuio.json_schema.JsonSchemaType] = {}

1475 defaults = {}

1476

1477 properties["$schema"] = yuio.json_schema.String()

1478

1479 for name, field in cls.__get_fields().items():

1480 if field.is_subconfig:

1481 properties[name] = field.ty.to_json_schema(ctx)

1482 else:

1483 assert field.parser

1484 field_schema = field.parser.to_json_schema(ctx)

1485 if field.help and field.help is not yuio.DISABLED:

1486 field_schema = yuio.json_schema.Meta(

1487 field_schema, description=field.help

1488 )

1489 properties[name] = field_schema

1490 if field.default is not yuio.MISSING:

1491 try:

1492 defaults[name] = field.parser.to_json_value(field.default)

1493 except TypeError:

1494 pass

1495

1496 return yuio.json_schema.Meta(

1497 yuio.json_schema.Object(properties),

1498 title=cls.__name__,

1499 description=cls.__doc__,

1500 default=defaults,

1501 )

1502

1503

1504Config.__init_subclass__(_allow_positionals=False)

Coverage for yuio / config.py: 94%

447 statements