.proteoform_simulation_metrics

proteopy.utils.proteoform_simulation_metrics(adata, *, score_key='proteoform_score', pval_key='proteoform_score_pval_adj', call_key='is_proteoform', cluster_key='cluster_id', verbose=False)[source]

Measure how well planted proteoform groups were recovered.

Takes one object yielded by proteoform_group_simulator() after a proteoform inference method has been run on it, and returns one record of summary metrics plus the per-protein results they are computed from.

Parameters:

  • adata (AnnData) – Simulated peptide-level AnnData with .var columns source_protein_id, simulation_type, score_key, pval_key, call_key and cluster_key.

  • score_key (str) – .var column with the per-protein proteoform score. Higher values rank a protein as more likely a proteoform.

  • pval_key (str) – .var column with the per-protein (adjusted) p-value. Lower values rank a protein as more likely a proteoform.

  • call_key (str) – .var column flagging proteins called as having proteoforms (truthy values; NaN counts as not called).

  • cluster_key (str) – .var column with the inferred peptide cluster.

  • verbose (bool) – If True, print the metrics.

Return type:

tuple[dict, DataFrame]

Returns:

  • metrics (dict) – Summary metrics. The simulation parameters are not included; they are in .uns['proteoform_simulation'] of adata.

    Metrics come at two levels, which differ in which proteins are positives (planted proteoform groups). All other proteins are background (pseudo-negatives), and a called protein is a predicted positive at both levels.

    • _call: mixed proteins are positives.

    • _cluster: mixed proteins whose inferred clusters match their source proteins exactly (ARI = 1) are positives. Mixed proteins with a wrong clustering are background, so calling them counts toward background_detection_rate_cluster. Which mixed proteins are positives depends on the clustering of the inference method.

    Background proteins are not certain negatives: unmixed proteins are real proteins and may carry real proteoforms. Metrics are therefore named after what they measure in the simulation rather than their classical classification terms.

    For each level (* = call or cluster):

    • detection_rate_*: fraction of positives called (true positive rate, recall)

    • background_detection_rate_*: fraction of background proteins called (false positive rate). An upper bound of the false positive rate, as some calls may be real proteoforms

    • score_auroc_*, score_auprc_*: area under the ROC and precision-recall curves, ranking proteins by score_key (higher first)

    • pval_auroc_*, pval_auprc_*: the same, ranking proteins by pval_key (lower first)

    and, independent of the level:

    • mean_ari: mean adjusted Rand index between the inferred clusters and the source proteins over called mixed proteins. Each cluster label, including the noise label, counts as its own group.

    • cluster_match_rate: fraction of called mixed proteins whose inferred clusters match their source proteins exactly (ARI = 1)

    Proteins without a score or p-value rank last. The ROC and precision-recall curves sweep a cutoff over the proteins of this one dataset, so they do not depend on the calling thresholds of the inference method. They treat background proteins as negatives, so real proteoforms among them lower both areas.

    Rates are NaN when undefined (e.g. no positives). The calls of most methods depend on multiple-testing correction and thereby on the mixed:unmixed ratio, as does AUPRC, whose chance level is the fraction of positives. AUROC does not depend on the ratio.

  • results (pandas.DataFrame) – One row per simulated protein, indexed by protein_id, with columns simulation_type, called (bool), score (from score_key), pval (from pval_key), ari (adjusted Rand index; NaN for unmixed proteins) and cluster_match (ARI = 1; False for unmixed proteins).

Raises:

KeyError – If a required .var column is missing.

Examples

>>> import proteopy as pr
>>> sim = next(pr.utils.proteoform_group_simulator(
...     adata, random_state=42,
... ))
>>> pr.tl.proteoform_group_inference(sim, method="copf")
>>> metrics, results = pr.utils.proteoform_simulation_metrics(sim)