Configuration#

The configuration class is defined in configuration.py. From CLI, you can load custom configuration yaml with -c or --config argument. From Python, you can replace the default configuration with

from pathlib import Path
import sec_certs.configuration as config_module

config_module.config.load_from_yaml("/path/to/your/config.yaml")

# or just set the individual key
config_module.config.log_filepath = Path("/some/path/where/log/will/be/stored.txt")

The configuration yaml is a simple flat dictionary of keys and values. The configuration file can specify only some of the fields. For the content of unspecified fields, environment variable with seccerts_ prefix (case insensitive) will be checked. If such variable is not set, default value will be used. Content in the yaml always beats the environment variable.

For instance, when user provides the following yaml

log_filepath: my_own_log_file.txt
n_threads: 7

and sets SECCERTS_MINIMAL_TOKEN_LENGTH=4 as environment variable, only these 3 keys will be loaded with config.load_from_yaml(), others will be untouched.

Tip

You can load settings even without providing yaml configuration. Simply set the corresponding environment variables or use .env file.

Configuration keys, types, default values and descriptions#

log_filepath

  • type: <class 'pathlib.Path'>

  • default: ./cert_processing_log.log

  • description: Path to the file, relative to working directory, where the log will be stored.

  • env name: SECCERTS_LOG_FILEPATH

always_false_positive_fips_cert_id_threshold

  • type: <class 'int'>

  • default: 40

  • description: During validation we don’t connect certificates with number lower than this to connections due to these numbers being typically false positives.

  • env name: SECCERTS_ALWAYS_FALSE_POSITIVE_FIPS_CERT_ID_THRESHOLD

year_difference_between_validations

  • type: <class 'int'>

  • default: 7

  • description: During validation we don’t connect certificates with validation dates difference higher than this.

  • env name: SECCERTS_YEAR_DIFFERENCE_BETWEEN_VALIDATIONS

n_threads

  • type: <class 'int'>

  • default: -1

  • description: How many threads to use for parallel computations. Set to -1 to use all logical cores.

  • env name: SECCERTS_N_THREADS

cpe_matching_threshold

  • type: <class 'int'>

  • default: 92

  • description: Level of required string similarity between CPE and certificate name on CC CPE matching, 0-100. Lower values yield more false negatives, higher values more false positives

  • env name: SECCERTS_CPE_MATCHING_THRESHOLD

cpe_n_max_matches

  • type: <class 'int'>

  • default: 99

  • description: Maximum number of candidate CPE items that may be related to given certificate, >0

  • env name: SECCERTS_CPE_N_MAX_MATCHES

cc_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/cc/dataset.json

  • description: URL from where to fetch the latest snapshot of fully processed CC dataset.

  • env name: SECCERTS_CC_LATEST_SNAPSHOT

cc_latest_full_archive

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/cc/cc.tar.gz

  • description: URL from where to fetch the latest full archive of fully processed CC dataset.

  • env name: SECCERTS_CC_LATEST_FULL_ARCHIVE

cc_maintenances_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/cc/maintenance_updates.json

  • description: URL from where to fetch the latest snapshot of CC maintenance updates

  • env name: SECCERTS_CC_MAINTENANCES_LATEST_SNAPSHOT

pp_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/static/pp.json

  • description: URL from where to fetch the latest snapshot of the PP dataset.

  • env name: SECCERTS_PP_LATEST_SNAPSHOT

fips_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/fips/dataset.json

  • description: URL for the latest snapshot of FIPS dataset.

  • env name: SECCERTS_FIPS_LATEST_SNAPSHOT

fips_latest_full_archive

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/fips/fips.tar.gz

  • description: URL from where to fetch the latest full archive of fully processed FIPS dataset.

  • env name: SECCERTS_FIPS_LATEST_FULL_ARCHIVE

fips_iut_dataset

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/fips/iut/dataset.json

  • description: URL for the dataset of FIPS IUT data.

  • env name: SECCERTS_FIPS_IUT_DATASET

fips_iut_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/fips/iut/latest.json

  • description: URL for the latest snapshot of FIPS IUT data.

  • env name: SECCERTS_FIPS_IUT_LATEST_SNAPSHOT

fips_mip_dataset

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/fips/mip/dataset.json

  • description: URL for the dataset of FIPS MIP data

  • env name: SECCERTS_FIPS_MIP_DATASET

fips_mip_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/fips/mip/latest.json

  • description: URL for the latest snapshot of FIPS MIP data

  • env name: SECCERTS_FIPS_MIP_LATEST_SNAPSHOT

cpe_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/vuln/cpe/cpe.json.gz

  • description: URL for the latest snapshot of CPEDataset.

  • env name: SECCERTS_CPE_LATEST_SNAPSHOT

cve_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/vuln/cve/cve.json.gz

  • description: URL for the latest snapshot of CVEDataset.

  • env name: SECCERTS_CVE_LATEST_SNAPSHOT

cpe_match_latest_snapshot

  • type: <class 'pydantic_core._pydantic_core.Url'>

  • default: https://sec-certs.org/vuln/cpe/cpe_match.json.gz

  • description: URL for the latest snapshot of cpe match json.

  • env name: SECCERTS_CPE_MATCH_LATEST_SNAPSHOT

fips_matching_threshold

  • type: <class 'int'>

  • default: 90

  • description: Level of required similarity before FIPS IUT/MIP entry is considered to match a FIPS certificate.

  • env name: SECCERTS_FIPS_MATCHING_THRESHOLD

minimal_token_length

  • type: <class 'int'>

  • default: 3

  • description: Minimal length of a string that will be considered as a token during keyword extraction in CVE matching

  • env name: SECCERTS_MINIMAL_TOKEN_LENGTH

ignore_first_page

  • type: <class 'bool'>

  • default: True

  • description: During keyword search, first page usually contains addresses - ignore it.

  • env name: SECCERTS_IGNORE_FIRST_PAGE

cc_reference_annotator_dir

  • type: typing.Optional[pathlib.Path]

  • default: None

  • description: Path to directory with serialized reference annotator model. If set to null, tool will search default directory for the given dataset.

  • env name: SECCERTS_CC_REFERENCE_ANNOTATOR_DIR

cc_reference_annotator_should_train

  • type: <class 'bool'>

  • default: True

  • description: True if new reference annotator model shall be build, False otherwise.

  • env name: SECCERTS_CC_REFERENCE_ANNOTATOR_SHOULD_TRAIN

cc_matching_threshold

  • type: <class 'int'>

  • default: 70

  • description: Level of required similarity before CC scheme entry is considered to match a CC certificate.

  • env name: SECCERTS_CC_MATCHING_THRESHOLD

cc_use_proxy

  • type: <class 'bool'>

  • default: False

  • description: Download CC artifacts through the sec-certs.org proxy.

  • env name: SECCERTS_CC_USE_PROXY

fips_use_proxy

  • type: <class 'bool'>

  • default: False

  • description: Download FIPS artifacts through the sec-certs.org proxy.

  • env name: SECCERTS_FIPS_USE_PROXY

enable_progress_bars

  • type: <class 'bool'>

  • default: True

  • description: If true, progress bars will be printed to stdout during computation.

  • env name: SECCERTS_ENABLE_PROGRESS_BARS

nvd_api_key

  • type: typing.Optional[str]

  • default: None

  • description: NVD API key for access to CVEs and CPEs.

  • env name: SECCERTS_NVD_API_KEY

preferred_source_nvd_datasets

  • type: typing.Literal['sec-certs', 'api']

  • default: sec-certs

  • description: If set to sec-certs, will fetch CPE and CVE datasets from sec-certs.org. If set to api, will fetch these resources from NVD API. It is advised to set an nvd_api_key when setting this to api.

  • env name: SECCERTS_PREFERRED_SOURCE_NVD_DATASETS