{"id":1179,"date":"2026-10-21T00:30:42","date_gmt":"2026-10-20T19:00:42","guid":{"rendered":"https:\/\/codeanddebug.in\/blog\/?p=1179"},"modified":"2026-10-21T00:30:43","modified_gmt":"2026-10-20T19:00:43","slug":"confusion-matrix-in-machine-learning","status":"publish","type":"post","link":"https:\/\/codeanddebug.in\/blog\/confusion-matrix-in-machine-learning\/","title":{"rendered":"Confusion Matrix in Machine Learning: An Interactive Guide"},"content":{"rendered":"\n<style>.cdai{--cd-brand:#3c3fde;--cd-brand-soft:#eceeff;--cd-ink:#14162b;--cd-muted:#5b5f7a;--cd-line:#dfe1ee;--cd-card:#ffffff;--cd-bg:#f6f7fc;--cd-good:#0f8a5f;--cd-good-soft:#e3f6ee;--cd-bad:#c2410c;--cd-bad-soft:#fdebe1;--cd-warn:#b7791f;--cd-on-brand:#ffffff;\nfont-family:Inter,system-ui,-apple-system,\"Segoe UI\",Roboto,sans-serif;color:var(--cd-ink);background:var(--cd-bg);border:1px solid var(--cd-line);border-radius:14px;padding:22px;margin:28px 0;line-height:1.55;font-size:16px;box-sizing:border-box;max-width:100%;overflow:hidden}\n.s-dark .cdai,.cdai.cd-dark{--cd-brand:#8d90ff;--cd-brand-soft:#23254a;--cd-ink:#e8e9f5;--cd-muted:#a3a6c2;--cd-line:#32354f;--cd-card:#1b1d33;--cd-bg:#14162a;--cd-good:#4ccf98;--cd-good-soft:#163a2c;--cd-bad:#ff9a6b;--cd-bad-soft:#3d2316;--cd-warn:#f2c46b;--cd-on-brand:#0f1030}\n.cdai *,.cdai *::before,.cdai *::after{box-sizing:border-box}\n.cdai h4.cd-h{font-family:Outfit,Inter,sans-serif;font-size:1.25rem;font-weight:700;margin:0 0 4px;color:var(--cd-ink);line-height:1.3}\n.cdai .cd-sub{color:var(--cd-muted);font-size:.95rem;margin:0 0 16px}\n.cdai .cd-tag{display:inline-block;font-size:.72rem;font-weight:700;letter-spacing:.06em;text-transform:uppercase;color:var(--cd-brand);background:var(--cd-brand-soft);padding:3px 9px;border-radius:99px;margin-bottom:10px}\n.cdai button.cd-btn,.cdai .cd-chip{font:inherit;font-size:.92rem;font-weight:600;cursor:pointer;border-radius:10px;border:1.5px solid var(--cd-line);background:var(--cd-card);color:var(--cd-ink);padding:9px 14px;line-height:1.3;transition:transform .15s ease,border-color .15s ease,background .15s ease;text-transform:none;letter-spacing:normal;box-shadow:none;margin:0;min-height:0;height:auto}\n.cdai button.cd-btn:hover,.cdai .cd-chip:hover{border-color:var(--cd-brand);transform:translateY(-1px)}\n.cdai button.cd-btn:focus-visible,.cdai .cd-chip:focus-visible,.cdai input:focus-visible{outline:3px solid var(--cd-brand);outline-offset:2px}\n.cdai .cd-primary,.cdai button.cd-primary{background:var(--cd-brand);border-color:var(--cd-brand);color:var(--cd-on-brand)}\n.cdai .cd-chip[aria-pressed=\"true\"]{background:var(--cd-brand);border-color:var(--cd-brand);color:var(--cd-on-brand)}\n.cdai .cd-row{display:flex;flex-wrap:wrap;gap:10px;align-items:center}\n.cdai .cd-grid2{display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,230px),1fr));gap:12px}\n.cdai .cd-card{background:var(--cd-card);border:1px solid var(--cd-line);border-radius:12px;padding:14px}\n.cdai .cd-stat{font-family:Outfit,Inter,sans-serif;font-size:2rem;font-weight:700;color:var(--cd-brand);line-height:1.1;font-variant-numeric:tabular-nums}\n.cdai .cd-small{font-size:.85rem;color:var(--cd-muted)}\n.cdai label.cd-lab{font-weight:600;font-size:.92rem;display:flex;justify-content:space-between;gap:10px;margin:14px 0 6px}\n.cdai input[type=range]{width:100%;accent-color:var(--cd-brand);margin:0;height:28px;background:transparent;border:0;padding:0}\n.cdai canvas{display:block;width:100%;height:auto;border-radius:10px;background:var(--cd-card);border:1px solid var(--cd-line);touch-action:pan-y}\n\/* hero *\/\n.cdai.cd-hero{padding:0;position:relative;background:#101233;border-color:#24275a}\n.cdai.cd-hero canvas{border:0;border-radius:0;background:transparent;height:200px}\n.cdai.cd-hero .cd-hero-txt{position:absolute;left:0;right:0;bottom:0;padding:16px 20px;background:linear-gradient(transparent,rgba(16,18,51,.92) 55%);color:#fff}\n.cdai.cd-hero .cd-hero-txt b{font-family:Outfit,Inter,sans-serif;font-size:1.15rem}\n.cdai.cd-hero .cd-hero-txt span{display:block;color:#c9cbff;font-size:.9rem}\n\/* planner *\/\n.cdai .cd-timeline{display:flex;flex-direction:column;gap:7px;margin-top:6px}\n.cdai .cd-tl-row{display:grid;grid-template-columns:minmax(0,150px) 1fr 62px;gap:10px;align-items:center;font-size:.88rem}\n.cdai .cd-tl-name{font-weight:600;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}\n.cdai .cd-tl-name a{color:inherit;text-decoration:none}\n.cdai .cd-tl-name a:hover{color:var(--cd-brand)}\n.cdai .cd-tl-track{height:14px;background:var(--cd-brand-soft);border-radius:99px;overflow:hidden}\n.cdai .cd-tl-bar{height:100%;background:var(--cd-brand);border-radius:99px;width:0;transition:width .6s cubic-bezier(.2,.8,.2,1)}\n.cdai .cd-tl-bar.cd-skim{background:repeating-linear-gradient(45deg,var(--cd-brand),var(--cd-brand) 6px,transparent 6px,transparent 10px)}\n.cdai .cd-tl-w{text-align:right;font-variant-numeric:tabular-nums;color:var(--cd-muted)}\n.cdai .cd-prog{height:10px;border-radius:99px;background:var(--cd-brand-soft);overflow:hidden;margin-top:8px}\n.cdai .cd-prog i{display:block;height:100%;background:var(--cd-good);width:0;transition:width .4s ease}\n@media (max-width:520px){.cdai{padding:16px}.cdai .cd-tl-row{grid-template-columns:minmax(0,110px) 1fr 50px;font-size:.8rem}}\n\/* game *\/\n.cdai .cd-game-card{background:var(--cd-card);border:1px solid var(--cd-line);border-radius:12px;padding:18px;min-height:96px;font-size:1.05rem;font-weight:600;display:flex;align-items:center}\n.cdai .cd-feedback{margin-top:12px;padding:12px 14px;border-radius:10px;font-size:.93rem}\n.cdai .cd-feedback.cd-ok{background:var(--cd-good-soft);color:var(--cd-ink);border-left:4px solid var(--cd-good)}\n.cdai .cd-feedback.cd-no{background:var(--cd-bad-soft);color:var(--cd-ink);border-left:4px solid var(--cd-bad)}\n.cdai .cd-venn{position:relative;width:min(100%,300px);aspect-ratio:1\/1;margin:6px auto 0}\n.cdai .cd-venn div{position:absolute;border-radius:50%;border:2px solid var(--cd-brand);display:flex;justify-content:center;font-weight:700;font-size:.8rem;color:var(--cd-ink);padding-top:6px;transition:background .3s ease,transform .3s ease}\n.cdai .cd-venn div.cd-lit{background:var(--cd-brand-soft);transform:scale(1.03)}\n.cdai .cd-v1{inset:0}.cdai .cd-v2{inset:12%}.cdai .cd-v3{inset:26%}.cdai .cd-v4{inset:40%;align-items:center;padding-top:0!important}\n\/* checklist *\/\n.cdai.cd-check{padding:16px 18px;background:var(--cd-card)}\n.cdai.cd-check ul{list-style:none;margin:8px 0 0;padding:0}\n.cdai.cd-check li{margin:0;padding:0}\n.cdai.cd-check label{display:flex;gap:10px;align-items:flex-start;padding:7px 4px;cursor:pointer;font-size:.95rem;border-radius:8px}\n.cdai.cd-check label:hover{background:var(--cd-bg)}\n.cdai.cd-check input{width:18px;height:18px;margin-top:3px;accent-color:var(--cd-good);flex:none}\n.cdai.cd-check .cd-done{color:var(--cd-good);font-weight:700}\n\/* tokens *\/\n.cdai .cd-bars{display:flex;flex-direction:column;gap:6px;margin-top:10px}\n.cdai .cd-bar-row{display:grid;grid-template-columns:110px 1fr 52px;gap:10px;align-items:center;font-size:.9rem}\n.cdai .cd-bar-row b{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-weight:600}\n.cdai .cd-bar-row .cd-tl-bar{transition:width .35s ease}\n.cdai .cd-out{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;background:var(--cd-card);border:1px solid var(--cd-line);border-radius:10px;padding:12px;margin-top:12px;min-height:46px;font-size:.92rem;word-break:break-word}\n.cdai .cd-out mark{background:var(--cd-brand-soft);color:var(--cd-brand);padding:1px 4px;border-radius:4px;animation:cdpop .4s ease}\n@keyframes cdpop{from{transform:scale(1.4);opacity:0}to{transform:none;opacity:1}}\n.cdai .cd-meter{font-family:Outfit,Inter,sans-serif;font-weight:700;font-size:1.6rem;font-variant-numeric:tabular-nums}\n@media (prefers-reduced-motion:reduce){.cdai *{transition:none!important;animation:none!important}}\n.cdai .cd-tbl-wrap{overflow-x:auto;margin-top:10px}\n.cdai table.cd-tbl{border-collapse:collapse;width:100%;font-size:.88rem;font-variant-numeric:tabular-nums;background:var(--cd-card);margin:0}\n.cdai table.cd-tbl th,.cdai table.cd-tbl td{border:1px solid var(--cd-line);padding:6px 8px;text-align:center;color:var(--cd-ink)}\n.cdai table.cd-tbl th{background:var(--cd-brand-soft);font-weight:700;font-size:.8rem}\n.cdai table.cd-tbl input{width:64px;font:inherit;padding:4px 6px;border:1.5px solid var(--cd-line);border-radius:6px;background:var(--cd-bg);color:var(--cd-ink);text-align:center;margin:0}\n.cdai .cd-steps{margin-top:12px;display:grid;gap:6px;font-size:.92rem}\n.cdai .cd-steps div{background:var(--cd-card);border:1px solid var(--cd-line);border-radius:8px;padding:8px 12px}\n.cdai .cd-steps code,.cdai .cd-eq{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:.9rem;background:transparent;color:var(--cd-ink);padding:0}\n.cdai .cd-eq{display:inline-block;background:var(--cd-brand-soft);color:var(--cd-brand);padding:4px 10px;border-radius:8px;font-weight:700}\n.cdai.cd-codebox{padding:0;background:#16182c;border-color:#2a2d48}\n.cdai.cd-codebox .cd-codehead{display:flex;justify-content:space-between;align-items:center;padding:8px 14px;border-bottom:1px solid #2a2d48;color:#a3a6c2;font-size:.8rem;font-weight:600}\n.cdai.cd-codebox pre{margin:0;padding:14px 16px;overflow-x:auto;background:transparent;border:0;color:#e6e7f2;font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:.86rem;line-height:1.6;white-space:pre}\n.cdai.cd-codebox pre code{background:transparent;color:inherit;padding:0;font:inherit}\n.cdai.cd-codebox .cd-copy{font:inherit;font-size:.78rem;font-weight:600;cursor:pointer;background:#2a2d48;color:#e6e7f2;border:0;border-radius:6px;padding:5px 10px;margin:0;text-transform:none;letter-spacing:normal}\n.cdai.cd-codebox .cd-outp{border-top:1px dashed #2a2d48;color:#8fe3b8}\n.cdai .cd-legend{display:flex;flex-wrap:wrap;gap:14px;font-size:.82rem;color:var(--cd-muted);margin-top:8px}\n.cdai .cd-legend i{display:inline-block;width:10px;height:10px;border-radius:50%;margin-right:5px;vertical-align:middle}\n.cdai canvas.cd-touch{touch-action:none;cursor:crosshair}\n.cdai canvas.cd-grab{touch-action:none;cursor:grab}\n.cdai .cd-q-opts{display:grid;gap:8px;margin-top:12px}\n.cdai .cd-q-opts button.cd-btn{text-align:left;width:100%}\n.cdai .cd-q-opts button.cd-right{border-color:var(--cd-good);background:var(--cd-good-soft)}\n.cdai .cd-q-opts button.cd-wrong{border-color:var(--cd-bad);background:var(--cd-bad-soft)}\n.cdai .cd-odds-dots{display:grid;grid-template-columns:repeat(10,1fr);gap:6px;max-width:320px;margin-top:10px}\n.cdai .cd-odds-dots i{display:block;aspect-ratio:1\/1;border-radius:50%;background:var(--cd-line);transition:background .25s ease}\n.cdai .cd-cm{display:grid;grid-template-columns:auto 1fr 1fr;gap:6px;font-size:.88rem;max-width:420px;margin-top:12px}\n.cdai .cd-cm div{padding:10px 8px;border-radius:8px;text-align:center}\n.cdai .cd-cm .cd-cm-h{color:var(--cd-muted);font-weight:600;font-size:.78rem;align-self:center}\n.cdai .cd-cm .cd-cm-c{background:var(--cd-card);border:1px solid var(--cd-line)}\n.cdai .cd-cm .cd-cm-c b{display:block;font-family:Outfit,Inter,sans-serif;font-size:1.5rem;font-variant-numeric:tabular-nums}\n.cdai .cd-cm .cd-cm-good{border-color:var(--cd-good);background:var(--cd-good-soft)}\n.cdai .cd-cm .cd-cm-bad{border-color:var(--cd-bad);background:var(--cd-bad-soft)}\n.cdai.cd-q{background:var(--cd-card);padding:18px 20px;margin:18px 0;border-left:4px solid var(--cd-brand)}\n.cdai.cd-q[hidden]{display:none!important}\n.cdai.cd-q.cd-is-known{border-left-color:var(--cd-good)}\n.cdai.cd-q .cd-q-meta{display:flex;flex-wrap:wrap;gap:6px;margin-bottom:6px}\n.cdai .cd-pill{font-size:.72rem;font-weight:700;padding:3px 9px;border-radius:99px;background:var(--cd-brand-soft);color:var(--cd-brand);letter-spacing:.02em}\n.cdai .cd-pill.cd-lv-Freshers{background:var(--cd-good-soft);color:var(--cd-good)}\n.cdai .cd-pill.cd-lv-Experienced{background:var(--cd-bad-soft);color:var(--cd-bad)}\n.cdai.cd-q h3.cd-q-title{font-family:Outfit,Inter,sans-serif;font-size:1.15rem;line-height:1.35;margin:4px 0 10px;color:var(--cd-ink)}\n.cdai.cd-q .cd-q-body{font-size:.97rem}\n.cdai.cd-q .cd-q-body p{margin:0 0 10px}\n.cdai.cd-q .cd-q-body ul,.cdai.cd-q .cd-q-body ol{margin:0 0 10px 1.2em;padding:0}\n.cdai.cd-q .cd-q-body li{margin:3px 0}\n.cdai.cd-q .cd-q-body code,.cdai.cd-fc .cd-fc-a code{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:.86em;background:var(--cd-brand-soft);color:var(--cd-ink);padding:1px 5px;border-radius:4px}\n.cdai.cd-q .cd-q-body pre code,.cdai.cd-fc .cd-fc-a pre code{background:transparent;color:inherit;padding:0;font-size:inherit}\n.cdai.cd-q .cd-q-body table,.cdai.cd-fc table{border-collapse:collapse;width:100%;font-size:.88rem;margin:6px 0 12px}\n.cdai.cd-q .cd-q-body th,.cdai.cd-q .cd-q-body td,.cdai.cd-fc th,.cdai.cd-fc td{border:1px solid var(--cd-line);padding:6px 8px;text-align:left;color:var(--cd-ink)}\n.cdai.cd-q .cd-q-body th,.cdai.cd-fc th{background:var(--cd-brand-soft)}\n.cdai.cd-q .cd-codebox,.cdai.cd-fc .cd-codebox{margin:12px 0 4px}\n.cdai.cd-q .cd-q-foot{display:flex;justify-content:space-between;align-items:center;gap:10px;margin-top:10px;flex-wrap:wrap}\n.cdai.cd-q .cd-known[aria-pressed=\"true\"]{background:var(--cd-good-soft);border-color:var(--cd-good);color:var(--cd-good)}\n.cdai.cd-q .cd-show{display:none}\n.cdai.cd-q.cd-practice:not(.cd-revealed) .cd-q-body{display:none}\n.cdai.cd-q.cd-practice:not(.cd-revealed) .cd-show{display:inline-block}\n.cdai .cd-q-top{font-size:.8rem;color:var(--cd-muted);text-decoration:none}\n.cdai.cd-pyq-tool .cd-stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(110px,1fr));gap:10px;margin:12px 0}\n.cdai.cd-pyq-tool select,.cdai.cd-pyq-tool input[type=search]{font:inherit;font-size:.92rem;padding:9px 12px;border:1.5px solid var(--cd-line);border-radius:10px;background:var(--cd-card);color:var(--cd-ink);margin:0;min-width:0;width:100%;height:auto}\n.cdai.cd-pyq-tool .cd-filters{display:grid;grid-template-columns:1fr 1fr;gap:10px;margin-top:12px}\n@media (max-width:560px){.cdai.cd-pyq-tool .cd-filters{grid-template-columns:1fr}}\n.cdai.cd-pyq-tool label.cd-tog{display:flex;align-items:center;gap:8px;font-size:.9rem;cursor:pointer}\n.cdai.cd-pyq-tool label.cd-tog input{width:18px;height:18px;accent-color:var(--cd-brand);margin:0}\n.cdai a.cd-btn{display:inline-block;text-decoration:none}\n.cdai a.cd-dl{background:var(--cd-good);border-color:var(--cd-good);color:#fff}\n.cdai.cd-fc{position:fixed;inset:0;z-index:99999;background:rgba(10,12,30,.72);display:flex;align-items:center;justify-content:center;margin:0;border:0;border-radius:0;padding:max(12px,env(safe-area-inset-top,0px)) max(12px,env(safe-area-inset-right,0px)) max(12px,env(safe-area-inset-bottom,0px)) max(12px,env(safe-area-inset-left,0px))}\n.cdai.cd-fc[hidden]{display:none}\n.cdai.cd-fc .cd-fc-card{background:var(--cd-card);color:var(--cd-ink);border-radius:16px;padding:20px;width:min(720px,100%);max-height:100%;overflow-y:auto;box-shadow:0 20px 60px rgba(0,0,0,.35)}\n.cdai.cd-fc .cd-fc-meta{display:flex;gap:6px;margin:12px 0 6px}\n.cdai.cd-fc .cd-fc-q{font-family:Outfit,Inter,sans-serif;font-size:1.3rem;font-weight:700;line-height:1.35;margin-bottom:14px}\n.cdai.cd-fc .cd-fc-a{border-top:1px dashed var(--cd-line);padding-top:12px;margin-bottom:12px;font-size:.95rem}\n.cdai.cd-fc .cd-fc-a p{margin:0 0 10px}.cdai.cd-fc .cd-fc-a ul,.cdai.cd-fc .cd-fc-a ol{margin:0 0 10px 1.2em;padding:0}\n.cdai.cd-fc .cd-fc-actions{margin-top:6px}\n.cdai a.cd-btn{font:inherit;font-size:.92rem;font-weight:600;cursor:pointer;border-radius:10px;border:1.5px solid var(--cd-line);background:var(--cd-card);color:var(--cd-ink);padding:9px 14px;line-height:1.3;text-decoration:none;display:inline-block;transition:transform .15s ease,border-color .15s ease}\n.cdai a.cd-btn:hover{transform:translateY(-1px);border-color:var(--cd-brand)}\n.cdai a.cd-btn.cd-dl{background:#0f8a5f;border-color:#0f8a5f;color:#fff}\n.cdai a.cd-btn.cd-dl:hover{background:#0c7550}\n.cdai a.cd-btn:focus-visible{outline:3px solid var(--cd-brand);outline-offset:2px}\n.cdai .cd-brief{background:var(--cd-card);border:1px solid var(--cd-line);border-left:5px solid var(--cd-brand);border-radius:12px;padding:16px 18px}\n.cdai .cd-brief h4{font-family:Outfit,Inter,sans-serif;margin:0 0 8px;font-size:1.1rem;color:var(--cd-ink)}\n.cdai .cd-brief ul{margin:0;padding-left:20px}.cdai .cd-brief li{margin:4px 0}\n.cdai .cd-pre{display:grid;gap:10px;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));margin-top:10px}\n.cdai .cd-pre a{display:block;text-decoration:none;color:var(--cd-ink);background:var(--cd-card);border:1px solid var(--cd-line);border-radius:12px;padding:12px 14px;transition:border-color .15s}\n.cdai .cd-pre a:hover{border-color:var(--cd-brand)}\n.cdai .cd-pre a b{display:block;color:var(--cd-brand);font-size:.95rem}\n.cdai .cd-pre a span{font-size:.85rem;color:var(--cd-muted)}\n.cdai .cd-map{display:grid;grid-template-columns:repeat(auto-fit,minmax(160px,1fr));gap:8px;margin-top:10px;font-size:.88rem}\n.cdai .cd-map div{background:var(--cd-card);border:1px solid var(--cd-line);border-radius:10px;padding:10px 12px}\n.cdai .cd-map b{color:var(--cd-brand)}\n.cdai .cd-mail{font-weight:600;font-size:1.02rem;margin-top:4px}\n.cdai .cd-pill-bad{background:var(--cd-bad-soft);color:var(--cd-bad)}\n.cdai .cd-pill-good{background:var(--cd-good-soft);color:var(--cd-good)}\n.cdai .cd-sort{display:grid;grid-template-columns:1.3fr 1fr;gap:16px;align-items:start}\n.cdai .cd-sort-opts{display:grid;grid-template-columns:1fr 1fr;gap:8px;margin-top:12px}\n.cdai .cd-sort-opts button.cd-right{border-color:var(--cd-good);background:var(--cd-good-soft)}\n.cdai .cd-sort-opts button.cd-wrong{border-color:var(--cd-bad);background:var(--cd-bad-soft)}\n.cdai button.cd-btn:disabled{cursor:default;transform:none;opacity:.85}\n.cdai .cd-cm4{display:grid;grid-template-columns:auto 1fr 1fr;gap:6px;font-size:.85rem}\n.cdai .cd-cm4>div{padding:10px 6px;border-radius:8px;text-align:center}\n.cdai .cd-cm4 .h{color:var(--cd-muted);font-weight:600;font-size:.76rem;align-self:center}\n.cdai .cd-cm4 .c{border:1.5px solid var(--cd-line);background:var(--cd-card)}\n.cdai .cd-cm4 .c b{display:block;font-family:Outfit,Inter,sans-serif;font-size:1.5rem;font-variant-numeric:tabular-nums}\n.cdai .cd-cm4 .g{border-color:var(--cd-good);background:var(--cd-good-soft)}\n.cdai .cd-cm4 .r{border-color:var(--cd-bad);background:var(--cd-bad-soft)}\n.cdai .cd-par{display:grid;grid-template-columns:1fr 1fr;gap:12px;margin-top:14px}\n.cdai .cd-par .cd-card dl{display:grid;grid-template-columns:1fr auto;gap:4px 10px;margin:8px 0 0;font-size:.9rem}\n.cdai .cd-par dd{margin:0;font-weight:700;font-variant-numeric:tabular-nums;text-align:right}\n.cdai .cd-par dt{color:var(--cd-muted)}\n.cdai input[type=range]{width:100%;accent-color:var(--cd-brand);margin:6px 0}\n.cdai .cd-calc-in{display:grid;grid-template-columns:auto 1fr 1fr;gap:6px;max-width:380px;font-size:.85rem;align-items:center}\n.cdai .cd-calc-in label{display:block;border-radius:8px;padding:8px;text-align:center;font-weight:600;border:1.5px solid var(--cd-line)}\n.cdai .cd-calc-in label.g{border-color:var(--cd-good);background:var(--cd-good-soft)}\n.cdai .cd-calc-in label.r{border-color:var(--cd-bad);background:var(--cd-bad-soft)}\n.cdai .cd-calc-in .h{color:var(--cd-muted);font-weight:600;font-size:.76rem;text-align:center}\n.cdai .cd-calc-in input{width:100%;max-width:90px;font:inherit;font-size:1.05rem;font-weight:700;text-align:center;padding:5px;border:1.5px solid var(--cd-line);border-radius:6px;background:var(--cd-card);color:var(--cd-ink);margin:4px 0 0}\n.cdai table.cd-mtab td{text-align:left;vertical-align:top}\n.cdai table.cd-mtab td code{font-size:.8rem;background:transparent;color:var(--cd-ink);padding:0;white-space:nowrap}\n.cdai table.cd-mtab td.cd-res-v{font-family:Outfit,Inter,sans-serif;font-weight:700;font-size:1.1rem;text-align:center;color:var(--cd-brand)}\n.cdai .cd-f1bars{display:grid;gap:10px;margin-top:12px}\n.cdai .cd-f1bars>div{display:grid;grid-template-columns:120px 1fr 56px;gap:10px;align-items:center;font-size:.9rem}\n.cdai .cd-f1bars .t{height:14px;background:var(--cd-brand-soft);border-radius:99px;overflow:hidden}\n.cdai .cd-f1bars .t i{display:block;height:100%;background:var(--cd-brand);border-radius:99px;transition:width .2s}\n.cdai .cd-f1bars .t.f i{background:var(--cd-good)}.cdai .cd-f1bars .t.b i{background:var(--cd-warn)}\n.cdai .cd-f1bars b{text-align:right;font-variant-numeric:tabular-nums}\n.cdai .cd-thr-grid{display:grid;grid-template-columns:1fr 1fr;gap:14px;margin-top:12px}\n.cdai canvas{display:block;width:100%;background:var(--cd-card);border:1px solid var(--cd-line);border-radius:10px}\n.cdai .cd-kpis{display:grid;grid-template-columns:repeat(5,1fr);gap:8px;margin-top:12px}\n.cdai .cd-kpis div{background:var(--cd-card);border:1px solid var(--cd-line);border-radius:10px;padding:8px;text-align:center;font-size:.76rem;color:var(--cd-muted)}\n.cdai .cd-kpis b{display:block;font-family:Outfit,Inter,sans-serif;font-size:1.15rem;color:var(--cd-ink);font-variant-numeric:tabular-nums}\n.cdai .cd-mc-grid{display:grid;grid-template-columns:auto repeat(3,1fr);gap:6px;font-size:.85rem;max-width:520px;margin-top:12px}\n.cdai .cd-mc-c{padding:10px 6px;border-radius:8px;text-align:center;border:1.5px solid var(--cd-line);background:var(--cd-card);font-family:Outfit,Inter,sans-serif;font-size:1.3rem;font-weight:700}\n.cdai .cd-mc-c.cd-mc-h{border:0;background:transparent;font-family:Inter,sans-serif;font-size:.74rem;font-weight:600;color:var(--cd-muted);align-self:center}\n.cdai .cd-mc-c.cd-k-tp{border-color:var(--cd-good);background:var(--cd-good-soft)}\n.cdai .cd-mc-c.cd-k-fn{border-color:var(--cd-bad);background:var(--cd-bad-soft)}\n.cdai .cd-mc-c.cd-k-fp{border-color:var(--cd-warn);background:rgba(183,121,31,.14)}\n.cdai .cd-mc-c.cd-k-tn{opacity:.7}\n.cdai .cd-mc-avg{display:grid;grid-template-columns:repeat(3,1fr);gap:10px;margin-top:12px}\n.cdai .cd-mc-avg .cd-meter{font-size:1.3rem}\n.cdai .cd-key{display:flex;flex-wrap:wrap;gap:12px;font-size:.8rem;color:var(--cd-muted);margin-top:10px}\n.cdai .cd-key i{display:inline-block;width:12px;height:12px;border-radius:3px;vertical-align:-1px;margin-right:4px;border:1.5px solid}\n@media (max-width:640px){.cdai{padding:16px}.cdai .cd-sort,.cdai .cd-par,.cdai .cd-thr-grid,.cdai .cd-mc-avg{grid-template-columns:1fr}.cdai .cd-kpis{grid-template-columns:repeat(3,1fr)}.cdai .cd-f1bars>div{grid-template-columns:90px 1fr 48px}.cdai .cd-sort-opts{grid-template-columns:1fr}}\n@media (max-width:640px){.cdai table.cd-mtab thead{display:none}.cdai table.cd-mtab,.cdai table.cd-mtab tbody{display:block}.cdai table.cd-mtab tr{display:grid;grid-template-columns:1fr auto;border:1px solid var(--cd-line);border-radius:10px;margin-bottom:8px;background:var(--cd-card);overflow:hidden}.cdai table.cd-mtab td{border:0;padding:6px 10px}.cdai table.cd-mtab td:nth-child(1){grid-column:1;grid-row:1}.cdai table.cd-mtab td:nth-child(4){grid-column:2;grid-row:1;align-self:center}.cdai table.cd-mtab td:nth-child(2),.cdai table.cd-mtab td:nth-child(3){grid-column:1\/3;padding-top:0}.cdai table.cd-mtab td code{white-space:normal}}\n<\/style>\n\n\n\n<p class=\"wp-block-paragraph\">Your model scores 99% accuracy. Your manager is happy, the notebook looks great, and then the model goes live and misses every single fraud case it was built to catch. This happens more often than people admit, and the confusion matrix is the tool that would have warned you before launch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This guide explains the confusion matrix in machine learning from zero. You will sort real predictions into the four boxes yourself, calculate every metric by hand with a live calculator, slide a threshold and watch the matrix, ROC curve and precision-recall curve change together, and finish with a 3-class matrix and scikit-learn code. By the end you should not need another tab open.<\/p>\n\n\n\n<div class=\"cdai\" style=\"padding:18px\"><div class=\"cd-brief\"><h4>The short version (read this first)<\/h4><ul>\n<li>A <b>confusion matrix<\/b> is a table that compares what a classification model <b>predicted<\/b> with what was <b>actually<\/b> true.<\/li>\n<li>For a yes\/no problem it has 4 boxes: <b>True Positive<\/b>, <b>False Positive<\/b>, <b>False Negative<\/b> and <b>True Negative<\/b>.<\/li>\n<li>Every popular classification metric comes from those 4 numbers: accuracy, precision, recall, specificity, F1 score, and the points on the ROC curve.<\/li>\n<li>Accuracy alone can lie when one class is rare. Precision tells you how trustworthy the positive predictions are; recall tells you how many real positives you caught.<\/li>\n<li>Which metric matters depends on which mistake costs more: a false alarm (false positive) or a miss (false negative).<\/li>\n<\/ul><\/div><\/div>\n\n\n\n<h2 id=\"cd-prereq\" class=\"wp-block-heading\">Before you start: what you need to know<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You do not need advanced maths. If you can divide one number by another, you can follow every formula here. These topics will make the guide easier, and each link opens our own detailed guide:<\/p>\n\n\n\n<div class=\"cdai\" style=\"padding:18px\"><span class=\"cd-tag\">Prerequisites<\/span><div class=\"cd-pre\">\n<a href=\"https:\/\/codeanddebug.in\/blog\/logistic-regression-in-machine-learning\/\"><b>Logistic regression<\/b><span>Most important. Explains how a classifier outputs a probability and how the 0.5 threshold turns it into a yes or no.<\/span><\/a>\n<a href=\"https:\/\/codeanddebug.in\/blog\/linear-regression-in-machine-learning\/\"><b>Linear regression<\/b><span>Helpful background on how a model learns from data and how we measure its error.<\/span><\/a>\n<a href=\"https:\/\/codeanddebug.in\/blog\/how-to-learn-ai-ml-from-scratch-roadmap\/\"><b>AI and ML roadmap<\/b><span>Shows where model evaluation fits in the full machine learning path.<\/span><\/a>\n<a href=\"https:\/\/codeanddebug.in\/blog\/pandas-python-roadmap\/\"><b>Pandas roadmap<\/b><span>Only needed for the Python section, to handle real datasets.<\/span><\/a>\n<a href=\"https:\/\/codeanddebug.in\/blog\/python-interview-questions\/\"><b>Python interview questions<\/b><span>Brush up on lists, loops and functions before running the code.<\/span><\/a>\n<\/div>\n<p class=\"cd-small\" style=\"margin:12px 0 0\"><b>New to all of this?<\/b> Read the first 6 sections, play with every widget, and skip the multi-class and Python parts for now. <b>CBSE Class 10 AI students:<\/b> your syllabus covers the 2\u00d72 matrix, accuracy, precision, recall and F1. Sections 1 to 6 cover all of it.<\/p><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Here is how the guide is laid out, so you can jump to what you need:<\/p>\n\n\n\n<div class=\"cdai\" style=\"padding:18px\"><div class=\"cd-map\">\n<div><b>1.<\/b> Why accuracy is not enough<\/div><div><b>2.<\/b> What a confusion matrix is<\/div><div><b>3.<\/b> TP, FP, FN and TN<\/div>\n<div><b>4.<\/b> Type I and Type II errors<\/div><div><b>5.<\/b> Metrics and formulas<\/div><div><b>6.<\/b> Precision vs recall and F1<\/div>\n<div><b>7.<\/b> Threshold, ROC and AUC<\/div><div><b>8.<\/b> Multi-class matrices<\/div><div><b>9.<\/b> Choosing the right metric<\/div>\n<div><b>10.<\/b> Python with scikit-learn<\/div><div><b>11.<\/b> Common mistakes<\/div><div><b>12.<\/b> Quiz and FAQ<\/div><\/div><\/div>\n\n\n\n<h2 id=\"cd-accuracy\" class=\"wp-block-heading\">1. Why accuracy is not enough<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Accuracy is the share of predictions that were correct. It is the first number everyone looks at, and on balanced data it works fine. The trouble starts when one class is rare.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Take credit card fraud. Imagine 10,000 transactions where only 100 are fraud (1%). A &#8220;model&#8221; that ignores the data and says <em>not fraud<\/em> for every transaction is right 9,900 times. That is 99% accuracy, from a model that has never caught a single fraud. This is called the <strong>accuracy paradox<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now compare it with a real model that catches 80% of fraud but wrongly flags 5% of normal transactions. Move the slider to change how rare fraud is:<\/p>\n\n\n\n<div class=\"cdai cd-cm-paradox\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">The accuracy paradox<\/h4><p class=\"cd-sub\">Lazy model: always predicts &#8220;not fraud&#8221;. Real model: catches 80% of fraud, flags 5% of normal payments.<\/p><label class=\"cd-small\" for=\"cdpar\">Share of transactions that are fraud: <b class=\"cd-par-rate\"><\/b><\/label><input id=\"cdpar\" type=\"range\" min=\"1\" max=\"500\" value=\"10\">\n<div class=\"cd-par\"><div class=\"cd-card cd-par-lazy\"><b>Lazy model<\/b><dl><dt>Accuracy<\/dt><dd class=\"cd-par-acc\"><\/dd><dt>Recall (fraud caught)<\/dt><dd class=\"cd-par-rec\"><\/dd><dt>Fraud caught<\/dt><dd class=\"cd-par-caught\"><\/dd><dt>False alarms<\/dt><dd class=\"cd-par-fa\"><\/dd><\/dl><\/div>\n<div class=\"cd-card cd-par-real\"><b>Real model<\/b><dl><dt>Accuracy<\/dt><dd class=\"cd-par-acc\"><\/dd><dt>Recall (fraud caught)<\/dt><dd class=\"cd-par-rec\"><\/dd><dt>Fraud caught<\/dt><dd class=\"cd-par-caught\"><\/dd><dt>False alarms<\/dt><dd class=\"cd-par-fa\"><\/dd><\/dl><\/div><\/div>\n<div class=\"cd-feedback cd-no cd-par-msg\"><\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Accuracy squashes all the model&#8217;s behaviour into one number and hides <em>which<\/em> mistakes it makes. To see them, we need to split the predictions into groups. That split is the confusion matrix.<\/p>\n\n\n\n<h2 id=\"cd-what\" class=\"wp-block-heading\">2. What is a confusion matrix?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A confusion matrix is a table that summarises how a classification model performed on a test set by counting how many predictions fell into each combination of <em>actual class<\/em> and <em>predicted class<\/em>. The name comes from the question it answers: which classes is the model confusing with each other?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For a binary (two-class) problem, one class is called <strong>positive<\/strong> and the other <strong>negative<\/strong>. &#8220;Positive&#8221; does not mean good. It simply means the class you are trying to detect: spam, fraud, disease, a customer who will leave. Here is the layout used by scikit-learn and by most textbooks:<\/p>\n\n\n\n<div class=\"cdai\" style=\"padding:18px\"><div class=\"cd-cm4\" style=\"max-width:460px\">\n<div class=\"h\"><\/div><div class=\"h\">Predicted: Negative<\/div><div class=\"h\">Predicted: Positive<\/div>\n<div class=\"h\">Actual: Negative<\/div><div class=\"c g\"><b>TN<\/b>True Negative<br><span class=\"cd-small\">correctly cleared<\/span><\/div><div class=\"c r\"><b>FP<\/b>False Positive<br><span class=\"cd-small\">false alarm<\/span><\/div>\n<div class=\"h\">Actual: Positive<\/div><div class=\"c r\"><b>FN<\/b>False Negative<br><span class=\"cd-small\">missed<\/span><\/div><div class=\"c g\"><b>TP<\/b>True Positive<br><span class=\"cd-small\">correctly caught<\/span><\/div>\n<\/div><p class=\"cd-small\" style=\"margin:12px 0 0\">Rows = actual class, columns = predicted class. Green cells are correct predictions (the diagonal); orange cells are mistakes.<\/p><\/div>\n\n\n\n<h3 id=\"cd-read\" class=\"wp-block-heading\">How to read a confusion matrix<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Read across a row<\/strong> to see what happened to all examples of one actual class. The Actual Positive row tells you how many real positives were caught (TP) and how many were missed (FN).<\/li>\n\n\n\n<li><strong>Read down a column<\/strong> to see what a prediction really means. The Predicted Positive column tells you, out of everything the model flagged, how many were real (TP) and how many were false alarms (FP).<\/li>\n\n\n\n<li><strong>The diagonal is good.<\/strong> Everything on the top-left to bottom-right diagonal is a correct prediction. A perfect model has zeros everywhere else.<\/li>\n\n\n\n<li><strong>Totals check your work.<\/strong> TP + FP + FN + TN always equals the number of test examples.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Watch out:<\/strong> not every source uses the same layout. Some books and websites put Actual Positive in the first row and TP in the top-left. Always read the axis labels before reading the numbers. In scikit-learn, rows are actual and columns are predicted, ordered by label value, so for labels 0 and 1 the TN sits top-left and TP bottom-right.<\/p>\n\n\n\n<h2 id=\"cd-outcomes\" class=\"wp-block-heading\">3. True positive, false positive, false negative, true negative<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The four names look confusing until you notice they are built from two words, and each word answers one question:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>The second word (Positive or Negative) is what the model predicted.<\/strong><\/li>\n\n\n\n<li><strong>The first word (True or False) says whether that prediction was correct.<\/strong><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">So a <em>False Negative<\/em> means the model said negative and was wrong, which means the case was actually positive. Using a spam filter where positive = spam:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Outcome<\/th><th>Model said<\/th><th>Reality<\/th><th>Spam filter example<\/th><th>Good or bad?<\/th><\/tr><\/thead><tbody><tr><td>True Positive (TP)<\/td><td>Spam<\/td><td>Spam<\/td><td>Lottery scam goes to the spam folder<\/td><td>Correct<\/td><\/tr><tr><td>True Negative (TN)<\/td><td>Not spam<\/td><td>Not spam<\/td><td>Your Swiggy receipt stays in the inbox<\/td><td>Correct<\/td><\/tr><tr><td>False Positive (FP)<\/td><td>Spam<\/td><td>Not spam<\/td><td>An interview invite is hidden in spam<\/td><td>False alarm<\/td><\/tr><tr><td>False Negative (FN)<\/td><td>Not spam<\/td><td>Spam<\/td><td>A phishing mail lands in your inbox<\/td><td>Miss<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now try it. Each email below shows what it actually was and what the model decided. Pick the right box, and watch the confusion matrix fill up:<\/p>\n\n\n\n<div class=\"cdai cd-cm-sort\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">Sort the predictions<\/h4><p class=\"cd-sub\">Positive = spam. Decide which of the four boxes each prediction belongs in.<\/p>\n<div class=\"cd-sort\"><div><div class=\"cd-game-card cd-sort-card\" style=\"display:block\"><\/div>\n<div class=\"cd-sort-opts\"><button type=\"button\" class=\"cd-btn cd-sort-opt\" data-k=\"TP\">True Positive<\/button><button type=\"button\" class=\"cd-btn cd-sort-opt\" data-k=\"FP\">False Positive<\/button><button type=\"button\" class=\"cd-btn cd-sort-opt\" data-k=\"FN\">False Negative<\/button><button type=\"button\" class=\"cd-btn cd-sort-opt\" data-k=\"TN\">True Negative<\/button><\/div>\n<div class=\"cd-feedback\" hidden><\/div><div style=\"margin-top:10px\"><button type=\"button\" class=\"cd-btn cd-primary cd-sort-next\" hidden>Next email<\/button><\/div><\/div>\n<div><p class=\"cd-small\" style=\"margin:0 0 6px\"><b>Your confusion matrix<\/b><\/p><div class=\"cd-cm4\">\n<div class=\"h\"><\/div><div class=\"h\">Pred: not spam<\/div><div class=\"h\">Pred: spam<\/div>\n<div class=\"h\">Actual: not spam<\/div><div class=\"c g\"><b class=\"cd-g-TN\">0<\/b>TN<\/div><div class=\"c r\"><b class=\"cd-g-FP\">0<\/b>FP<\/div>\n<div class=\"h\">Actual: spam<\/div><div class=\"c r\"><b class=\"cd-g-FN\">0<\/b>FN<\/div><div class=\"c g\"><b class=\"cd-g-TP\">0<\/b>TP<\/div><\/div><\/div><\/div><\/div>\n\n\n\n<h2 id=\"cd-errors\" class=\"wp-block-heading\">4. Type I and Type II errors<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Statistics has older names for the two kinds of mistakes, and you will meet them in exams and interviews:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Type I error = False Positive.<\/strong> You raised an alarm that was not needed. A healthy person is told they might have a disease.<\/li>\n\n\n\n<li><strong>Type II error = False Negative.<\/strong> You missed something real. A sick person is told they are healthy.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A memory trick: the boy who cried wolf first made a Type I error (the villagers believed there was a wolf when there was none), and later the villagers made a Type II error (they believed there was no wolf when there was one).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The important point is that the two errors almost never cost the same. Missing a cancer is much worse than ordering one extra biopsy. Blocking a genuine payment annoys a customer; letting fraud through loses money. Keep that idea in mind, because the rest of this guide is about measuring and balancing these two costs.<\/p>\n\n\n\n<h2 id=\"cd-metrics\" class=\"wp-block-heading\">5. Metrics from the confusion matrix (with formulas)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Let us use a small, concrete example. A spam filter is tested on 12 emails. 5 are really spam and 7 are not. The results are: <strong>TP = 4, FP = 2, FN = 1, TN = 5<\/strong>. Each metric below answers a different question about these four numbers.<\/p>\n\n\n\n<h3 id=\"cd-acc\" class=\"wp-block-heading\">Accuracy<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">Accuracy = (TP + TN) \/ (TP + TN + FP + FN)<\/span><span class=\"cd-small\">(4 + 5) \/ 12 = 0.75<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">What fraction of all predictions were right? Simple and useful when classes are balanced and both errors cost the same. Misleading on imbalanced data, as you saw earlier.<\/p>\n\n\n\n<h3 id=\"cd-precision\" class=\"wp-block-heading\">Precision<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">Precision = TP \/ (TP + FP)<\/span><span class=\"cd-small\">4 \/ (4 + 2) = 0.667<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Of all the emails the model flagged as spam, how many really were spam? Precision is about <em>trust<\/em>: when the model says positive, how often is it right? Low precision means many false alarms. It is also called <strong>positive predictive value<\/strong>.<\/p>\n\n\n\n<h3 id=\"cd-recall\" class=\"wp-block-heading\">Recall (sensitivity, true positive rate)<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">Recall = TP \/ (TP + FN)<\/span><span class=\"cd-small\">4 \/ (4 + 1) = 0.8<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Of all the real spam emails, how many did the model catch? Recall is about <em>coverage<\/em>. Low recall means many misses. Doctors call it <strong>sensitivity<\/strong>, and on the ROC curve it is called the <strong>true positive rate (TPR)<\/strong>. All three names mean the same formula.<\/p>\n\n\n\n<h3 id=\"cd-specificity\" class=\"wp-block-heading\">Specificity (true negative rate)<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">Specificity = TN \/ (TN + FP)<\/span><span class=\"cd-small\">5 \/ (5 + 2) = 0.714<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Of all the genuine emails, how many were correctly left alone? Specificity is recall for the negative class. Medical tests usually report sensitivity and specificity together.<\/p>\n\n\n\n<h3 id=\"cd-fpr\" class=\"wp-block-heading\">False positive rate<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">FPR = FP \/ (FP + TN) = 1 \u2212 Specificity<\/span><span class=\"cd-small\">2 \/ (2 + 5) = 0.286<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Of all the genuine emails, how many were wrongly flagged? You will need this for the ROC curve.<\/p>\n\n\n\n<h3 id=\"cd-f1\" class=\"wp-block-heading\">F1 score<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">F1 = 2 \u00d7 Precision \u00d7 Recall \/ (Precision + Recall)<\/span><span class=\"cd-small\">2 \u00d7 0.667 \u00d7 0.8 \/ (0.667 + 0.8) = 0.727<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">A single score that is high only when <em>both<\/em> precision and recall are high. It is the harmonic mean of the two, which we explain in the next section.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now calculate them yourself. Type any four numbers, or load a preset, and every formula is worked out step by step:<\/p>\n\n\n\n<div class=\"cdai cd-cm-calc\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">Confusion matrix calculator<\/h4><p class=\"cd-sub\">Change any cell. Each metric shows its formula, your numbers plugged in, and the result.<\/p><div class=\"cd-row\" style=\"margin-bottom:12px\"><button type=\"button\" class=\"cd-chip cd-preset\" data-p=\"spam\" aria-pressed=\"true\">Spam example<\/button><button type=\"button\" class=\"cd-chip cd-preset\" data-p=\"fraud\">Imbalanced fraud<\/button><button type=\"button\" class=\"cd-chip cd-preset\" data-p=\"perfect\">Perfect model<\/button><button type=\"button\" class=\"cd-chip cd-preset\" data-p=\"all\">Flags everything<\/button><\/div>\n<div class=\"cd-calc-in\"><div class=\"h\"><\/div><div class=\"h\">Predicted negative<\/div><div class=\"h\">Predicted positive<\/div>\n<div class=\"h\">Actual negative<\/div><label class=\"g\">TN<input class=\"cd-in-tn\" type=\"number\" min=\"0\" value=\"5\" aria-label=\"True negatives\"><\/label><label class=\"r\">FP<input class=\"cd-in-fp\" type=\"number\" min=\"0\" value=\"2\" aria-label=\"False positives\"><\/label>\n<div class=\"h\">Actual positive<\/div><label class=\"r\">FN<input class=\"cd-in-fn\" type=\"number\" min=\"0\" value=\"1\" aria-label=\"False negatives\"><\/label><label class=\"g\">TP<input class=\"cd-in-tp\" type=\"number\" min=\"0\" value=\"4\" aria-label=\"True positives\"><\/label><\/div>\n<p class=\"cd-small cd-calc-err\" style=\"margin:10px 0 0\"><\/p><div class=\"cd-calc-out\"><\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Try the <em>Imbalanced fraud<\/em> preset: accuracy is 95.3%, yet precision is only 15.1% because the model raised 45 false alarms to catch 8 frauds. Then try <em>Flags everything<\/em>: recall is a perfect 100%, but precision drops to 10%. A model can always get perfect recall by calling everything positive, which is why you never report recall alone.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Metric<\/th><th>Formula<\/th><th>Question it answers<\/th><th>Also called<\/th><\/tr><\/thead><tbody><tr><td>Accuracy<\/td><td>(TP + TN) \/ Total<\/td><td>How often is the model right overall?<\/td><td>None<\/td><\/tr><tr><td>Precision<\/td><td>TP \/ (TP + FP)<\/td><td>When it says positive, how often is it right?<\/td><td>Positive predictive value<\/td><\/tr><tr><td>Recall<\/td><td>TP \/ (TP + FN)<\/td><td>How many real positives did it find?<\/td><td>Sensitivity, TPR, hit rate<\/td><\/tr><tr><td>Specificity<\/td><td>TN \/ (TN + FP)<\/td><td>How many real negatives did it clear?<\/td><td>True negative rate<\/td><\/tr><tr><td>FPR<\/td><td>FP \/ (FP + TN)<\/td><td>How many negatives were false alarms?<\/td><td>Fall-out, 1 \u2212 specificity<\/td><\/tr><tr><td>F1 score<\/td><td>2PR \/ (P + R)<\/td><td>Is it good at both precision and recall?<\/td><td>F-measure, Dice score<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 id=\"cd-tradeoff\" class=\"wp-block-heading\">6. Precision vs recall, and why F1 uses the harmonic mean<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In practice you can rarely push precision and recall up together. If you make the spam filter stricter, it flags fewer emails: the ones it does flag are almost surely spam (precision goes up), but more spam slips through (recall goes down). Make it more relaxed and the opposite happens. This is the <strong>precision-recall trade-off<\/strong>, and you will control it directly in the next section.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Because of this trade-off, people want one number that rewards a good balance. Why not just average them? Because a simple average can be fooled. A model with precision 1.0 and recall 0.02 (it flags one email, correctly, and misses the rest) has an average of 0.51, which sounds acceptable for a useless filter. The harmonic mean stays close to the smaller number, so F1 for that model is only about 0.04.<\/p>\n\n\n\n<div class=\"cdai cd-cm-f1\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">F1 score vs simple average<\/h4><p class=\"cd-sub\">Move precision and recall apart and compare the three scores. Beta controls the F-beta score.<\/p><label class=\"cd-small\">Precision: <b class=\"cd-f1-pv\"><\/b><\/label><input class=\"cd-f1-p\" type=\"range\" min=\"0\" max=\"100\" value=\"90\" aria-label=\"Precision\">\n<label class=\"cd-small\">Recall: <b class=\"cd-f1-rv\"><\/b><\/label><input class=\"cd-f1-r\" type=\"range\" min=\"0\" max=\"100\" value=\"20\" aria-label=\"Recall\">\n<label class=\"cd-small\">Beta for F-beta: <b class=\"cd-f1-bv\"><\/b><\/label><input class=\"cd-f1-b\" type=\"range\" min=\"0.5\" max=\"3\" step=\"0.5\" value=\"2\" aria-label=\"Beta\">\n<div class=\"cd-f1bars\"><div><span>Simple average<\/span><span class=\"t\"><i class=\"cd-f1-mean-bar\"><\/i><\/span><b class=\"cd-f1-mean-v\"><\/b><\/div>\n<div><span>F1 score<\/span><span class=\"t f\"><i class=\"cd-f1-f1-bar\"><\/i><\/span><b class=\"cd-f1-f1-v\"><\/b><\/div>\n<div><span class=\"cd-f1-fb-l\">F-beta<\/span><span class=\"t b\"><i class=\"cd-f1-fb-bar\"><\/i><\/span><b class=\"cd-f1-fb-v\"><\/b><\/div><\/div>\n<p class=\"cd-small cd-f1-msg\" style=\"margin:12px 0 0\"><\/p><\/div>\n\n\n\n<h3 id=\"cd-fbeta\" class=\"wp-block-heading\">F-beta score<\/h3>\n\n\n\n<div class=\"cdai\" style=\"padding:16px;display:grid;gap:10px;justify-items:center\"><span class=\"cd-eq\">F<sub>\u03b2<\/sub> = (1 + \u03b2\u00b2) \u00d7 P \u00d7 R \/ (\u03b2\u00b2 \u00d7 P + R)<\/span><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">F1 treats precision and recall as equally important. When they are not, use F-beta. With <strong>\u03b2 = 2<\/strong> (F2), recall counts more, which suits disease screening or fraud. With <strong>\u03b2 = 0.5<\/strong>, precision counts more, which suits spam filters and recommendations. F1 is simply the case \u03b2 = 1.<\/p>\n\n\n\n<h2 id=\"cd-roc\" class=\"wp-block-heading\">7. The threshold, the ROC curve and AUC<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Most classifiers do not output yes or no directly. They output a score or probability, like &#8220;0.83 chance of spam&#8221;, and a <strong>threshold<\/strong> (0.5 by default) turns it into a label. We covered this in the <a href=\"https:\/\/codeanddebug.in\/blog\/logistic-regression-in-machine-learning\/\">logistic regression guide<\/a>. The key insight: <strong>the confusion matrix is not fixed. Change the threshold and you get a different matrix<\/strong>, with different precision and recall.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the explorer below, each dot is one email placed at the score the model gave it. Orange dots are real spam, green dots are genuine. Everything to the right of the black line is predicted spam. Drag the threshold, then change the model quality:<\/p>\n\n\n\n<div class=\"cdai cd-cm-thr\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">Threshold, ROC and precision-recall explorer<\/h4><p class=\"cd-sub\">60 emails: 20 spam, 40 genuine. The red dot on each curve marks your current threshold.<\/p><canvas aria-label=\"Model scores of 60 emails with the threshold line\"><\/canvas>\n<div class=\"cd-grid2\" style=\"margin-top:10px\"><div><label class=\"cd-small\">Threshold: <b class=\"cd-t-thv\"><\/b><\/label><input class=\"cd-t-th\" type=\"range\" min=\"0.01\" max=\"0.99\" step=\"0.01\" value=\"0.5\" aria-label=\"Threshold\"><\/div>\n<div><label class=\"cd-small\">Model quality (class separation): <b class=\"cd-t-sepv\">good<\/b><\/label><input class=\"cd-t-sep\" type=\"range\" min=\"0\" max=\"3.5\" step=\"0.1\" value=\"1.6\" aria-label=\"Model quality\"><\/div><\/div>\n<div class=\"cd-kpis\"><div><b class=\"cd-t-tp\"><\/b>TP<\/div><div><b class=\"cd-t-fp\"><\/b>FP<\/div><div><b class=\"cd-t-fn\"><\/b>FN<\/div><div><b class=\"cd-t-tn\"><\/b>TN<\/div><div><b class=\"cd-t-auc\"><\/b>ROC AUC<\/div>\n<div><b class=\"cd-t-prec\"><\/b>Precision<\/div><div><b class=\"cd-t-rec\"><\/b>Recall<\/div><div><b class=\"cd-t-fpr\"><\/b>FPR<\/div><div><b class=\"cd-t-f1\"><\/b>F1<\/div><div><b class=\"cd-t-acc\"><\/b>Accuracy<\/div><\/div>\n<div class=\"cd-thr-grid\"><div><p class=\"cd-small\" style=\"margin:0 0 4px\"><b>ROC curve<\/b><\/p><canvas aria-label=\"ROC curve\"><\/canvas><\/div><div><p class=\"cd-small\" style=\"margin:0 0 4px\"><b>Precision-recall curve<\/b><\/p><canvas aria-label=\"Precision-recall curve\"><\/canvas><\/div><\/div>\n<p class=\"cd-small cd-t-msg\" style=\"margin:12px 0 0\"><\/p><\/div>\n\n\n\n<h3 id=\"cd-roc-what\" class=\"wp-block-heading\">What is the ROC curve?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">ROC stands for <strong>Receiver Operating Characteristic<\/strong>, a name from radar engineering in World War II. The ROC curve plots the true positive rate (recall) on the y-axis against the false positive rate on the x-axis, with one point for every possible threshold. Each point is one confusion matrix.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The bottom-left corner (0, 0) is a threshold so high that nothing is flagged.<\/li>\n\n\n\n<li>The top-right corner (1, 1) is a threshold so low that everything is flagged.<\/li>\n\n\n\n<li>The top-left corner (0, 1) is the dream: every positive caught, no false alarms.<\/li>\n\n\n\n<li>The dashed diagonal is a model that guesses at random.<\/li>\n<\/ul>\n\n\n\n<h3 id=\"cd-auc\" class=\"wp-block-heading\">What is AUC?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">AUC means <strong>Area Under the Curve<\/strong>, usually the area under the ROC curve. It ranges from 0 to 1 and summarises the model across all thresholds in one number. The cleanest way to understand it: AUC is the probability that the model gives a randomly chosen positive a higher score than a randomly chosen negative.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>ROC AUC<\/th><th>What it means<\/th><\/tr><\/thead><tbody><tr><td>1.0<\/td><td>Perfect ranking: every positive scores above every negative<\/td><\/tr><tr><td>0.9 to 1.0<\/td><td>Excellent<\/td><\/tr><tr><td>0.8 to 0.9<\/td><td>Good<\/td><\/tr><tr><td>0.7 to 0.8<\/td><td>Fair, often usable<\/td><\/tr><tr><td>0.5<\/td><td>No better than random guessing<\/td><\/tr><tr><td>Below 0.5<\/td><td>Worse than random; the labels may be flipped<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Set model quality to &#8220;random&#8221; in the explorer and the curve collapses towards the diagonal with AUC near 0.5. Push it to &#8220;excellent&#8221; and the curve hugs the top-left corner.<\/p>\n\n\n\n<h3 id=\"cd-pr\" class=\"wp-block-heading\">When to use the precision-recall curve instead<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The ROC curve uses the false positive rate, and on heavily imbalanced data that rate stays tiny even when the model makes many false alarms, because the number of negatives is huge. The ROC curve can therefore look excellent while precision is poor. When positives are rare (fraud, rare diseases, defects), look at the <strong>precision-recall curve<\/strong> and its summary, <strong>average precision<\/strong>. A random model&#8217;s PR curve sits at the positive rate, not at 0.5, so it is a much stricter test.<\/p>\n\n\n\n<h2 id=\"cd-multiclass\" class=\"wp-block-heading\">8. Confusion matrix for multi-class classification (3\u00d73 and beyond)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With more than two classes, the matrix grows to N \u00d7 N. A support-ticket classifier that routes tickets to Billing, Technical or Account teams gives a 3\u00d73 matrix. Rows are still actual classes, columns are still predicted classes, and the diagonal is still correct predictions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There is no single &#8220;positive&#8221; class any more, so you calculate metrics <em>one class at a time<\/em>, treating that class as positive and all others as negative (this is called one-vs-rest). For the chosen class:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>TP<\/strong> is its diagonal cell.<\/li>\n\n\n\n<li><strong>FN<\/strong> is the rest of its row: tickets of this class sent to another team.<\/li>\n\n\n\n<li><strong>FP<\/strong> is the rest of its column: other tickets wrongly sent to this team.<\/li>\n\n\n\n<li><strong>TN<\/strong> is everything else.<\/li>\n<\/ul>\n\n\n\n<div class=\"cdai cd-cm-multi\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">Explore a 3\u00d73 confusion matrix<\/h4><p class=\"cd-sub\">30 support tickets. Pick a class to treat as positive and see where its TP, FN and FP come from.<\/p><div class=\"cd-row\"><button type=\"button\" class=\"cd-chip\">Billing<\/button><button type=\"button\" class=\"cd-chip\">Technical<\/button><button type=\"button\" class=\"cd-chip\">Account<\/button><\/div>\n<div class=\"cd-mc-grid\"><\/div>\n<div class=\"cd-key\"><span><i style=\"border-color:var(--cd-good);background:var(--cd-good-soft)\"><\/i>TP<\/span><span><i style=\"border-color:var(--cd-bad);background:var(--cd-bad-soft)\"><\/i>FN (its row)<\/span><span><i style=\"border-color:var(--cd-warn)\"><\/i>FP (its column)<\/span><span><i style=\"border-color:var(--cd-line)\"><\/i>TN<\/span><\/div>\n<p class=\"cd-mc-detail\" style=\"font-size:.92rem;margin:12px 0 0\"><\/p><div class=\"cd-mc-avg\"><\/div><\/div>\n\n\n\n<h3 id=\"cd-averages\" class=\"wp-block-heading\">Macro, micro and weighted averages<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Once you have precision, recall and F1 per class, you combine them into one number in one of three ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Macro average:<\/strong> a simple average across classes. Every class counts equally, so a small class that performs badly drags the score down. Use it when every class matters equally.<\/li>\n\n\n\n<li><strong>Weighted average:<\/strong> each class is weighted by its <em>support<\/em> (number of true examples). It reflects overall performance but can hide a weak minority class.<\/li>\n\n\n\n<li><strong>Micro average:<\/strong> pools all TP, FP and FN across classes and then computes the metric. For single-label multi-class problems, micro F1 equals accuracy.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In the ticket example, the Account class has the lowest recall (0.625). Macro F1 (0.726) is slightly lower than weighted F1 (0.732) because the weaker Account class gets a full one-third of the vote in the macro average.<\/p>\n\n\n\n<h2 id=\"cd-which\" class=\"wp-block-heading\">9. Which metric should you use?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">There is no best metric in general, only the best metric for your problem. Ask one question: <strong>which mistake hurts more, a false positive or a false negative?<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Situation<\/th><th>Focus on<\/th><th>Why<\/th><\/tr><\/thead><tbody><tr><td>Missing a positive is dangerous (disease, fraud, security)<\/td><td>Recall<\/td><td>False negatives are the costly error<\/td><\/tr><tr><td>False alarms are costly (spam filter, legal action, recommendations)<\/td><td>Precision<\/td><td>False positives are the costly error<\/td><\/tr><tr><td>Both errors matter and classes are imbalanced<\/td><td>F1 or PR curve<\/td><td>Balances both; ignores the easy true negatives<\/td><\/tr><tr><td>Balanced classes, equal costs<\/td><td>Accuracy<\/td><td>Simple and honest in this case<\/td><\/tr><tr><td>Comparing models before choosing a threshold<\/td><td>ROC AUC<\/td><td>Threshold-independent ranking quality<\/td><\/tr><tr><td>Several classes, all equally important<\/td><td>Macro F1<\/td><td>Small classes are not ignored<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Test your judgement on six real scenarios:<\/p>\n\n\n\n<div class=\"cdai cd-cm-which\"><span class=\"cd-tag\">Try it<\/span><h4 class=\"cd-h\">Pick the right metric<\/h4><div class=\"cd-wh-body\"><p class=\"cd-sub\">Six real-world scenarios. Choose the metric you would optimise for each.<\/p><button type=\"button\" class=\"cd-btn cd-primary cd-wh-start\">Start<\/button><\/div><\/div>\n\n\n\n<h2 id=\"cd-python\" class=\"wp-block-heading\">10. Confusion matrix in Python with scikit-learn<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Everything above takes only a few lines in scikit-learn. Every output below is real, produced by running the code. If you are new to Python, our <a href=\"https:\/\/codeanddebug.in\/blog\/python-interview-questions\/\">Python interview questions guide<\/a> covers the basics used here.<\/p>\n\n\n\n<h3 id=\"cd-py-cm\" class=\"wp-block-heading\">Create a confusion matrix<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The same 12 spam emails from section 5. Note that <code>ravel()<\/code> returns the cells in the order TN, FP, FN, TP for labels 0 and 1.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#D4D4D4;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#1E1E1E\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#D4D4D4;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>from sklearn.metrics import confusion_matrix\n\n# 1 = spam, 0 = not spam, for 12 emails\nactual    = &#91;1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0&#93;\npredicted = &#91;1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1&#93;\n\ncm = confusion_matrix(actual, predicted)\nprint(cm)\n\ntn, fp, fn, tp = cm.ravel()   # sklearn order for binary labels 0 and 1\nprint(f\"TP={tp}  FP={fp}  FN={fn}  TN={tn}\")<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.metrics <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> confusion_matrix<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A9955\"># 1 = spam, 0 = not spam, for 12 emails<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">actual    = &#91;<\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">predicted = &#91;<\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">&#93;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">cm = confusion_matrix(actual, predicted)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(cm)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">tn, fp, fn, tp = cm.ravel()   <\/span><span style=\"color: #6A9955\"># sklearn order for binary labels 0 and 1<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #569CD6\">f<\/span><span style=\"color: #CE9178\">&quot;TP=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">tp<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\">  FP=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">fp<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\">  FN=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">fn<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\">  TN=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">tn<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\">&quot;<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Output:<\/strong><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D4D4D4\">[[5 2]<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"> [1 4]]<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">TP=4  FP=2  FN=1  TN=5<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 id=\"cd-py-metrics\" class=\"wp-block-heading\">Calculate all the metrics<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>classification_report<\/code> prints precision, recall and F1 for each class, plus macro and weighted averages. <em>Support<\/em> is the number of true examples of each class.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#D4D4D4;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#1E1E1E\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#D4D4D4;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>from sklearn.metrics import (accuracy_score, precision_score,\n                             recall_score, f1_score, classification_report)\n\nactual    = &#91;1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0&#93;\npredicted = &#91;1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1&#93;\n\nprint(\"Accuracy :\", round(accuracy_score(actual, predicted), 3))\nprint(\"Precision:\", round(precision_score(actual, predicted), 3))\nprint(\"Recall   :\", round(recall_score(actual, predicted), 3))\nprint(\"F1 score :\", round(f1_score(actual, predicted), 3))\nprint()\nprint(classification_report(actual, predicted, target_names=&#91;\"not spam\", \"spam\"&#93;))<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.metrics <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> (accuracy_score, precision_score,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">                             recall_score, f1_score, classification_report)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">actual    = &#91;<\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">predicted = &#91;<\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">&#93;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;Accuracy :&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">round<\/span><span style=\"color: #D4D4D4\">(accuracy_score(actual, predicted), <\/span><span style=\"color: #B5CEA8\">3<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;Precision:&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">round<\/span><span style=\"color: #D4D4D4\">(precision_score(actual, predicted), <\/span><span style=\"color: #B5CEA8\">3<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;Recall   :&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">round<\/span><span style=\"color: #D4D4D4\">(recall_score(actual, predicted), <\/span><span style=\"color: #B5CEA8\">3<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;F1 score :&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">round<\/span><span style=\"color: #D4D4D4\">(f1_score(actual, predicted), <\/span><span style=\"color: #B5CEA8\">3<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(classification_report(actual, predicted, <\/span><span style=\"color: #9CDCFE\">target_names<\/span><span style=\"color: #D4D4D4\">=&#91;<\/span><span style=\"color: #CE9178\">&quot;not spam&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #CE9178\">&quot;spam&quot;<\/span><span style=\"color: #D4D4D4\">&#93;))<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Output:<\/strong><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D4D4D4\">Accuracy : 0.75<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">Precision: 0.667<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">Recall   : 0.8<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">F1 score : 0.727<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"><\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">              precision    recall  f1-score   support<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"><\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    not spam       0.83      0.71      0.77         7<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">        spam       0.67      0.80      0.73         5<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"><\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    accuracy                           0.75        12<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">   macro avg       0.75      0.76      0.75        12<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">weighted avg       0.76      0.75      0.75        12<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 id=\"cd-py-threshold\" class=\"wp-block-heading\">Change the threshold on a real dataset<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the trade-off on real medical data: the breast cancer dataset that ships with scikit-learn. We relabel it so that 1 means malignant, the class we must not miss, train a logistic regression, and compare two thresholds.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#D4D4D4;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#1E1E1E\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#D4D4D4;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>import numpy as np\nfrom sklearn.datasets import load_breast_cancer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import confusion_matrix, recall_score, precision_score\n\nX, y = load_breast_cancer(return_X_y=True)\ny = 1 - y          # make 1 = malignant (the class we must not miss)\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.3, random_state=7, stratify=y)\n\nmodel = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))\nmodel.fit(X_train, y_train)\nproba = model.predict_proba(X_test)&#91;:, 1&#93;     # probability of malignant\n\nfor threshold in &#91;0.5, 0.2&#93;:\n    pred = (proba >= threshold).astype(int)\n    tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()\n    print(f\"threshold {threshold}: TP={tp} FN={fn} FP={fp} TN={tn} | \"\n          f\"recall={recall_score(y_test, pred):.3f} precision={precision_score(y_test, pred):.3f}\")<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> numpy <\/span><span style=\"color: #C586C0\">as<\/span><span style=\"color: #D4D4D4\"> np<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.datasets <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> load_breast_cancer<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.model_selection <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> train_test_split<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.pipeline <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> make_pipeline<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.preprocessing <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> StandardScaler<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.linear_model <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> LogisticRegression<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.metrics <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> confusion_matrix, recall_score, precision_score<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">X, y = load_breast_cancer(<\/span><span style=\"color: #9CDCFE\">return_X_y<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #569CD6\">True<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">y = <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\"> - y          <\/span><span style=\"color: #6A9955\"># make 1 = malignant (the class we must not miss)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">X_train, X_test, y_train, y_test = train_test_split(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    X, y, <\/span><span style=\"color: #9CDCFE\">test_size<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">0.3<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #9CDCFE\">random_state<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">7<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #9CDCFE\">stratify<\/span><span style=\"color: #D4D4D4\">=y)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">model = make_pipeline(StandardScaler(), LogisticRegression(<\/span><span style=\"color: #9CDCFE\">max_iter<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">1000<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">model.fit(X_train, y_train)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">proba = model.predict_proba(X_test)&#91;:, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">&#93;     <\/span><span style=\"color: #6A9955\"># probability of malignant<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">for<\/span><span style=\"color: #D4D4D4\"> threshold <\/span><span style=\"color: #C586C0\">in<\/span><span style=\"color: #D4D4D4\"> &#91;<\/span><span style=\"color: #B5CEA8\">0.5<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #B5CEA8\">0.2<\/span><span style=\"color: #D4D4D4\">&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    pred = (proba &gt;= threshold).astype(<\/span><span style=\"color: #4EC9B0\">int<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    <\/span><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #569CD6\">f<\/span><span style=\"color: #CE9178\">&quot;threshold <\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">threshold<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\">: TP=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">tp<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\"> FN=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">fn<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\"> FP=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">fp<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\"> TN=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">tn<\/span><span style=\"color: #569CD6\">}<\/span><span style=\"color: #CE9178\"> | &quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">          <\/span><span style=\"color: #569CD6\">f<\/span><span style=\"color: #CE9178\">&quot;recall=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">recall_score(y_test, pred)<\/span><span style=\"color: #569CD6\">:.3f}<\/span><span style=\"color: #CE9178\"> precision=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">precision_score(y_test, pred)<\/span><span style=\"color: #569CD6\">:.3f}<\/span><span style=\"color: #CE9178\">&quot;<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Output:<\/strong><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D4D4D4\">threshold 0.5: TP=61 FN=3 FP=1 TN=106 | recall=0.953 precision=0.984<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">threshold 0.2: TP=62 FN=2 FP=4 TN=103 | recall=0.969 precision=0.939<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Lowering the threshold from 0.5 to 0.2 catches one more malignant tumour (FN drops from 3 to 2) at the cost of three more false alarms (FP rises from 1 to 4). For cancer screening that is usually a trade worth making; for a spam filter it might not be.<\/p>\n\n\n\n<h3 id=\"cd-py-roc\" class=\"wp-block-heading\">ROC AUC and average precision<\/h3>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#D4D4D4;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#1E1E1E\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#D4D4D4;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>from sklearn.datasets import load_breast_cancer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import roc_auc_score, roc_curve, average_precision_score\n\nX, y = load_breast_cancer(return_X_y=True)\ny = 1 - y\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=0.3, random_state=7, stratify=y)\nmodel = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)\nproba = model.predict_proba(X_test)&#91;:, 1&#93;\n\nprint(\"ROC AUC          :\", round(roc_auc_score(y_test, proba), 4))\nprint(\"Average precision:\", round(average_precision_score(y_test, proba), 4))\n\nfpr, tpr, thresholds = roc_curve(y_test, proba)\nprint(\"Points on the ROC curve:\", len(fpr))\nfor f, t, th in list(zip(fpr, tpr, thresholds))&#91;1:6&#93;:\n    print(f\"  threshold={th:.3f}  FPR={f:.3f}  TPR={t:.3f}\")<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.datasets <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> load_breast_cancer<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.model_selection <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> train_test_split<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.pipeline <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> make_pipeline<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.preprocessing <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> StandardScaler<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.linear_model <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> LogisticRegression<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.metrics <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> roc_auc_score, roc_curve, average_precision_score<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">X, y = load_breast_cancer(<\/span><span style=\"color: #9CDCFE\">return_X_y<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #569CD6\">True<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">y = <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\"> - y<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">X_train, X_test, y_train, y_test = train_test_split(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    X, y, <\/span><span style=\"color: #9CDCFE\">test_size<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">0.3<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #9CDCFE\">random_state<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">7<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #9CDCFE\">stratify<\/span><span style=\"color: #D4D4D4\">=y)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">model = make_pipeline(StandardScaler(), LogisticRegression(<\/span><span style=\"color: #9CDCFE\">max_iter<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">1000<\/span><span style=\"color: #D4D4D4\">)).fit(X_train, y_train)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">proba = model.predict_proba(X_test)&#91;:, <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">&#93;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;ROC AUC          :&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">round<\/span><span style=\"color: #D4D4D4\">(roc_auc_score(y_test, proba), <\/span><span style=\"color: #B5CEA8\">4<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;Average precision:&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">round<\/span><span style=\"color: #D4D4D4\">(average_precision_score(y_test, proba), <\/span><span style=\"color: #B5CEA8\">4<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">fpr, tpr, thresholds = roc_curve(y_test, proba)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #CE9178\">&quot;Points on the ROC curve:&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #DCDCAA\">len<\/span><span style=\"color: #D4D4D4\">(fpr))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #C586C0\">for<\/span><span style=\"color: #D4D4D4\"> f, t, th <\/span><span style=\"color: #C586C0\">in<\/span><span style=\"color: #D4D4D4\"> <\/span><span style=\"color: #4EC9B0\">list<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #DCDCAA\">zip<\/span><span style=\"color: #D4D4D4\">(fpr, tpr, thresholds))&#91;<\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\">:<\/span><span style=\"color: #B5CEA8\">6<\/span><span style=\"color: #D4D4D4\">&#93;:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    <\/span><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(<\/span><span style=\"color: #569CD6\">f<\/span><span style=\"color: #CE9178\">&quot;  threshold=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">th<\/span><span style=\"color: #569CD6\">:.3f}<\/span><span style=\"color: #CE9178\">  FPR=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">f<\/span><span style=\"color: #569CD6\">:.3f}<\/span><span style=\"color: #CE9178\">  TPR=<\/span><span style=\"color: #569CD6\">{<\/span><span style=\"color: #D4D4D4\">t<\/span><span style=\"color: #569CD6\">:.3f}<\/span><span style=\"color: #CE9178\">&quot;<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Output:<\/strong><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D4D4D4\">ROC AUC          : 0.9968<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">Average precision: 0.9951<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">Points on the ROC curve: 10<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">  threshold=1.000  FPR=0.000  TPR=0.016<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">  threshold=0.708  FPR=0.000  TPR=0.906<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">  threshold=0.646  FPR=0.009  TPR=0.906<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">  threshold=0.473  FPR=0.009  TPR=0.969<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">  threshold=0.123  FPR=0.075  TPR=0.969<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Each row is one threshold and its point on the ROC curve. To draw the curves, use <code>RocCurveDisplay.from_predictions(y_test, proba)<\/code> and <code>PrecisionRecallDisplay.from_predictions(y_test, proba)<\/code> from <code>sklearn.metrics<\/code>. To draw the matrix itself as a heatmap, use <code>ConfusionMatrixDisplay.from_predictions(y_test, pred)<\/code>.<\/p>\n\n\n\n<h3 id=\"cd-py-multi\" class=\"wp-block-heading\">Multi-class confusion matrix<\/h3>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#D4D4D4;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#1E1E1E\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" style=\"color:#D4D4D4;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>from sklearn.metrics import confusion_matrix, classification_report\n\n# Support tickets classified into 3 teams\nlabels = &#91;\"Billing\", \"Technical\", \"Account\"&#93;\nactual = (&#91;\"Billing\"&#93; * 10) + (&#91;\"Technical\"&#93; * 12) + (&#91;\"Account\"&#93; * 8)\npredicted = (&#91;\"Billing\"&#93; * 8 + &#91;\"Technical\"&#93; * 1 + &#91;\"Account\"&#93; * 1 +\n             &#91;\"Billing\"&#93; * 2 + &#91;\"Technical\"&#93; * 9 + &#91;\"Account\"&#93; * 1 +\n             &#91;\"Billing\"&#93; * 1 + &#91;\"Technical\"&#93; * 2 + &#91;\"Account\"&#93; * 5)\n\nprint(confusion_matrix(actual, predicted, labels=labels))\nprint()\nprint(classification_report(actual, predicted, labels=labels, digits=3))<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #C586C0\">from<\/span><span style=\"color: #D4D4D4\"> sklearn.metrics <\/span><span style=\"color: #C586C0\">import<\/span><span style=\"color: #D4D4D4\"> confusion_matrix, classification_report<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A9955\"># Support tickets classified into 3 teams<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">labels = &#91;<\/span><span style=\"color: #CE9178\">&quot;Billing&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #CE9178\">&quot;Technical&quot;<\/span><span style=\"color: #D4D4D4\">, <\/span><span style=\"color: #CE9178\">&quot;Account&quot;<\/span><span style=\"color: #D4D4D4\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">actual = (&#91;<\/span><span style=\"color: #CE9178\">&quot;Billing&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">10<\/span><span style=\"color: #D4D4D4\">) + (&#91;<\/span><span style=\"color: #CE9178\">&quot;Technical&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">12<\/span><span style=\"color: #D4D4D4\">) + (&#91;<\/span><span style=\"color: #CE9178\">&quot;Account&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">8<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">predicted = (&#91;<\/span><span style=\"color: #CE9178\">&quot;Billing&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">8<\/span><span style=\"color: #D4D4D4\"> + &#91;<\/span><span style=\"color: #CE9178\">&quot;Technical&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\"> + &#91;<\/span><span style=\"color: #CE9178\">&quot;Account&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\"> +<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">             &#91;<\/span><span style=\"color: #CE9178\">&quot;Billing&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">2<\/span><span style=\"color: #D4D4D4\"> + &#91;<\/span><span style=\"color: #CE9178\">&quot;Technical&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">9<\/span><span style=\"color: #D4D4D4\"> + &#91;<\/span><span style=\"color: #CE9178\">&quot;Account&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\"> +<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">             &#91;<\/span><span style=\"color: #CE9178\">&quot;Billing&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">1<\/span><span style=\"color: #D4D4D4\"> + &#91;<\/span><span style=\"color: #CE9178\">&quot;Technical&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">2<\/span><span style=\"color: #D4D4D4\"> + &#91;<\/span><span style=\"color: #CE9178\">&quot;Account&quot;<\/span><span style=\"color: #D4D4D4\">&#93; * <\/span><span style=\"color: #B5CEA8\">5<\/span><span style=\"color: #D4D4D4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(confusion_matrix(actual, predicted, <\/span><span style=\"color: #9CDCFE\">labels<\/span><span style=\"color: #D4D4D4\">=labels))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #DCDCAA\">print<\/span><span style=\"color: #D4D4D4\">(classification_report(actual, predicted, <\/span><span style=\"color: #9CDCFE\">labels<\/span><span style=\"color: #D4D4D4\">=labels, <\/span><span style=\"color: #9CDCFE\">digits<\/span><span style=\"color: #D4D4D4\">=<\/span><span style=\"color: #B5CEA8\">3<\/span><span style=\"color: #D4D4D4\">))<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Output:<\/strong><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.5rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><pre class=\"shiki dark-plus\" style=\"background-color: #1E1E1E\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D4D4D4\">[[8 1 1]<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"> [2 9 1]<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"> [1 2 5]]<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"><\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">              precision    recall  f1-score   support<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"><\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">     Billing      0.727     0.800     0.762        10<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">   Technical      0.750     0.750     0.750        12<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">     Account      0.714     0.625     0.667         8<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\"><\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">    accuracy                          0.733        30<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">   macro avg      0.731     0.725     0.726        30<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D4D4D4\">weighted avg      0.733     0.733     0.732        30<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">These are the same numbers you explored in the 3\u00d73 widget. Pass <code>labels=<\/code> so the rows and columns come out in the order you expect.<\/p>\n\n\n\n<h2 id=\"cd-mistakes\" class=\"wp-block-heading\">11. Common mistakes to avoid<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Reporting accuracy on imbalanced data.<\/strong> Always check the class balance first and show precision and recall alongside accuracy.<\/li>\n\n\n\n<li><strong>Mixing up the layout.<\/strong> Check whether rows are actual or predicted before reading any cell. Swapping them swaps precision and recall.<\/li>\n\n\n\n<li><strong>Forgetting which class is positive.<\/strong> In scikit-learn, binary metrics treat label 1 as positive by default. If your class of interest is 0, set <code>pos_label<\/code> or relabel.<\/li>\n\n\n\n<li><strong>Evaluating on training data.<\/strong> A confusion matrix on data the model has already seen is far too optimistic. Use a held-out test set or cross-validation.<\/li>\n\n\n\n<li><strong>Tuning the threshold on the test set.<\/strong> Choose the threshold on a validation set, then report the final matrix on the untouched test set.<\/li>\n\n\n\n<li><strong>Quoting recall without precision<\/strong> (or the reverse). Either one alone can be pushed to 100% by a useless model.<\/li>\n<\/ul>\n\n\n\n<h2 id=\"cd-quiz\" class=\"wp-block-heading\">12. Test yourself<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Six questions covering the whole guide. Five or more correct and you can explain a confusion matrix in any interview.<\/p>\n\n\n\n<div class=\"cdai cd-cm-quiz\"><span class=\"cd-tag\">Quiz<\/span><h4 class=\"cd-h\">Confusion matrix quiz<\/h4><div class=\"cd-cq-body\"><p class=\"cd-sub\">Takes about two minutes. Keep a calculator handy.<\/p><button type=\"button\" class=\"cd-btn cd-primary cd-cq-start\">Start the quiz<\/button><\/div><\/div>\n\n\n\n<h2 id=\"cd-faq\" class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What is a confusion matrix in machine learning?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A confusion matrix is a table that compares a classification model&#8217;s predictions with the actual labels. For two classes it has four cells: true positives, false positives, false negatives and true negatives. Metrics such as accuracy, precision, recall and F1 score are calculated from these four numbers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why is it called a confusion matrix?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Because it shows which classes the model confuses with each other. Every off-diagonal cell counts examples of one class that the model mistook for another.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is the difference between a false positive and a false negative?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A false positive is when the model predicts the positive class but the example is actually negative, such as a genuine email marked as spam. A false negative is when the model predicts negative but the example is actually positive, such as a spam email reaching the inbox.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is the formula for precision and recall?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Precision = TP \/ (TP + FP) and recall = TP \/ (TP + FN). Precision measures how many predicted positives were correct, and recall measures how many actual positives were found.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is a good F1 score?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">It depends on the problem and the baseline. As a rough guide, above 0.9 is excellent, 0.8 to 0.9 is good, and 0.5 to 0.8 is moderate. Always compare it against a simple baseline model on the same data.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is the full form of ROC and AUC?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">ROC stands for Receiver Operating Characteristic and AUC stands for Area Under the Curve. ROC AUC is the area under the ROC curve, which plots the true positive rate against the false positive rate at every threshold.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can a confusion matrix be used for multi-class classification?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Yes. For N classes it becomes an N \u00d7 N table. Metrics are calculated per class by treating each class as positive and the rest as negative, then combined with a macro, micro or weighted average.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Is accuracy a good metric for imbalanced datasets?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. When one class is rare, a model that always predicts the majority class gets high accuracy while being useless. Use precision, recall, F1 score or the precision-recall curve instead.<\/p>\n\n\n\n<script type=\"application\/ld+json\">{\"@context\": \"https:\/\/schema.org\", \"@type\": \"FAQPage\", \"mainEntity\": [{\"@type\": \"Question\", \"name\": \"What is a confusion matrix in machine learning?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"A confusion matrix is a table that compares a classification model's predictions with the actual labels. For two classes it has four cells: true positives, false positives, false negatives and true negatives. Metrics such as accuracy, precision, recall and F1 score are calculated from these four numbers.\"}}, {\"@type\": \"Question\", \"name\": \"Why is it called a confusion matrix?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"Because it shows which classes the model confuses with each other. Every off-diagonal cell counts examples of one class that the model mistook for another.\"}}, {\"@type\": \"Question\", \"name\": \"What is the difference between a false positive and a false negative?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"A false positive is when the model predicts the positive class but the example is actually negative, such as a genuine email marked as spam. A false negative is when the model predicts negative but the example is actually positive, such as a spam email reaching the inbox.\"}}, {\"@type\": \"Question\", \"name\": \"What is the formula for precision and recall?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"Precision = TP \/ (TP + FP) and recall = TP \/ (TP + FN). Precision measures how many predicted positives were correct, and recall measures how many actual positives were found.\"}}, {\"@type\": \"Question\", \"name\": \"What is a good F1 score?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"It depends on the problem and the baseline. As a rough guide, above 0.9 is excellent, 0.8 to 0.9 is good, and 0.5 to 0.8 is moderate. Always compare it against a simple baseline model on the same data.\"}}, {\"@type\": \"Question\", \"name\": \"What is the full form of ROC and AUC?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"ROC stands for Receiver Operating Characteristic and AUC stands for Area Under the Curve. ROC AUC is the area under the ROC curve, which plots the true positive rate against the false positive rate at every threshold.\"}}, {\"@type\": \"Question\", \"name\": \"Can a confusion matrix be used for multi-class classification?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"Yes. For N classes it becomes an N \u00d7 N table. Metrics are calculated per class by treating each class as positive and the rest as negative, then combined with a macro, micro or weighted average.\"}}, {\"@type\": \"Question\", \"name\": \"Is accuracy a good metric for imbalanced datasets?\", \"acceptedAnswer\": {\"@type\": \"Answer\", \"text\": \"No. When one class is rare, a model that always predicts the majority class gets high accuracy while being useless. Use precision, recall, F1 score or the precision-recall curve instead.\"}}]}<\/script>\n\n\n\n<h2 id=\"cd-next\" class=\"wp-block-heading\">What to learn next<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can now read any confusion matrix, calculate every metric by hand, and choose a threshold for the job. Next, learn cross-validation so your matrix reflects real performance, then models that output better probabilities, such as random forests and gradient boosting. Our <a href=\"https:\/\/codeanddebug.in\/blog\/how-to-learn-ai-ml-from-scratch-roadmap\/\">AI and ML roadmap<\/a> shows the full path, and the <a href=\"https:\/\/codeanddebug.in\/blog\/logistic-regression-in-machine-learning\/\">logistic regression guide<\/a> is the best place to practise everything you learned here on a real classifier.<\/p>\n\n\n\n<script src=\"data:text\/javascript;charset=utf-8;base64,KGZ1bmN0aW9uKCl7CiJ1c2Ugc3RyaWN0IjsKZnVuY3Rpb24gJChzLHIpe3JldHVybiAocnx8ZG9jdW1lbnQpLnF1ZXJ5U2VsZWN0b3Iocyl9CmZ1bmN0aW9uICQkKHMscil7cmV0dXJuIEFycmF5LnByb3RvdHlwZS5zbGljZS5jYWxsKChyfHxkb2N1bWVudCkucXVlcnlTZWxlY3RvckFsbChzKSl9CmZ1bmN0aW9uIGNzcyhlbCxuKXtyZXR1cm4gZ2V0Q29tcHV0ZWRTdHlsZShlbCkuZ2V0UHJvcGVydHlWYWx1ZShuKS50cmltKCl9CmZ1bmN0aW9uIGN2MihjdixoKXt2YXIgZD1NYXRoLm1pbih3aW5kb3cuZGV2aWNlUGl4ZWxSYXRpb3x8MSwyKSx3PWN2LmNsaWVudFdpZHRofHw1MDA7Y3Yud2lkdGg9dypkO2N2LmhlaWdodD1oKmQ7Y3Yuc3R5bGUuaGVpZ2h0PWgrInB4Ijt2YXIgYz1jdi5nZXRDb250ZXh0KCIyZCIpO2Muc2V0VHJhbnNmb3JtKGQsMCwwLGQsMCwwKTtyZXR1cm4ge2M6Yyx3OncsaDpofX0KZnVuY3Rpb24gY29sKGVsKXtyZXR1cm4ge2luazpjc3MoZWwsIi0tY2QtaW5rIiksbXV0OmNzcyhlbCwiLS1jZC1tdXRlZCIpLGxpbmU6Y3NzKGVsLCItLWNkLWxpbmUiKSxicjpjc3MoZWwsIi0tY2QtYnJhbmQiKSxiYWQ6Y3NzKGVsLCItLWNkLWJhZCIpLGdvb2Q6Y3NzKGVsLCItLWNkLWdvb2QiKSxjYXJkOmNzcyhlbCwiLS1jZC1jYXJkIil9fQpmdW5jdGlvbiBwY3Qodil7cmV0dXJuIGlzRmluaXRlKHYpPyh2KjEwMCkudG9GaXhlZCgxKSsiJSI6InVuZGVmaW5lZCJ9CmZ1bmN0aW9uIGYzKHYpe3JldHVybiBpc0Zpbml0ZSh2KT92LnRvRml4ZWQoMyk6InVuZGVmaW5lZCJ9CmZ1bmN0aW9uIHJuZyhzZWVkKXt2YXIgcz1zZWVkPj4+MDtyZXR1cm4gZnVuY3Rpb24oKXtzPShzKjE2NjQ1MjUrMTAxMzkwNDIyMyk+Pj4wO3JldHVybiBzLzQyOTQ5NjcyOTZ9fQpmdW5jdGlvbiBnYXVzcyhyKXt2YXIgdT0wLHY9MDt3aGlsZSh1PT09MCl1PXIoKTt3aGlsZSh2PT09MCl2PXIoKTtyZXR1cm4gTWF0aC5zcXJ0KC0yKk1hdGgubG9nKHUpKSpNYXRoLmNvcygyKk1hdGguUEkqdil9CmZ1bmN0aW9uIG1ldHJpY3ModHAsZnAsZm4sdG4pe3ZhciBuPXRwK2ZwK2ZuK3RuLHA9dHAvKHRwK2ZwKSxyPXRwLyh0cCtmbik7cmV0dXJuIHtuOm4sYWNjOih0cCt0bikvbixwcmVjOnAscmVjOnIsc3BlYzp0bi8odG4rZnApLGZwcjpmcC8oZnArdG4pLGZucjpmbi8oZm4rdHApLGYxOjIqcCpyLyhwK3IpfX0KCi8qIDEuIFNvcnQgdGhlIHByZWRpY3Rpb25zICovCiQkKCIuY2RhaS5jZC1jbS1zb3J0IikuZm9yRWFjaChmdW5jdGlvbihib3gpewogdmFyIEU9WwogIHtzOiJDb25ncmF0dWxhdGlvbnMhIFlvdSB3b24gUnMgMTAsMDAsMDAwLiBDbGFpbSBub3ciLGE6MSxwOjF9LAogIHtzOiJZb3VyIFN3aWdneSBvcmRlciBoYXMgYmVlbiBkZWxpdmVyZWQiLGE6MCxwOjB9LAogIHtzOiJJbnRlcnZpZXcgaW52aXRlOiBEYXRhIEFuYWx5c3Qgcm9sZSBhdCBJbmZvc3lzIixhOjAscDoxfSwKICB7czoiVVJHRU5UOiB5b3VyIGJhbmsgS1lDIGV4cGlyZXMgdG9kYXksIGNsaWNrIGxpbmsiLGE6MSxwOjB9LAogIHtzOiJMb3dlc3QgcHJpY2UgaVBob25lLCBvbmx5IHRvZGF5LCA5MCUgb2ZmISEhIixhOjEscDoxfSwKICB7czoiVGVhbSBtZWV0aW5nIG1vdmVkIHRvIDMgcG0iLGE6MCxwOjB9LAogIHtzOiJZb3VyIE9UUCBmb3IgbG9naW4gaXMgNDgyOTEzIixhOjAscDoxfSwKICB7czoiV29yayBmcm9tIGhvbWUgYW5kIGVhcm4gUnMgNTAsMDAwIHBlciBkYXkiLGE6MSxwOjB9XTsKIHZhciBMPXtUUDoiVHJ1ZSBQb3NpdGl2ZSIsRlA6IkZhbHNlIFBvc2l0aXZlIixGTjoiRmFsc2UgTmVnYXRpdmUiLFROOiJUcnVlIE5lZ2F0aXZlIn07CiBmdW5jdGlvbiB0cnV0aChlKXtyZXR1cm4gZS5wPyhlLmE\/IlRQIjoiRlAiKTooZS5hPyJGTiI6IlROIil9CiB2YXIgaT0wLHNjb3JlPTAsY291bnRzPXtUUDowLEZQOjAsRk46MCxUTjowfSxjYXJkPSQoIi5jZC1zb3J0LWNhcmQiLGJveCksZmI9JCgiLmNkLWZlZWRiYWNrIixib3gpLG54PSQoIi5jZC1zb3J0LW5leHQiLGJveCksYnRucz0kJCgiLmNkLXNvcnQtb3B0Iixib3gpOwogZnVuY3Rpb24gZ3JpZCgpe1siVFAiLCJGUCIsIkZOIiwiVE4iXS5mb3JFYWNoKGZ1bmN0aW9uKGspeyQoIi5jZC1nLSIrayxib3gpLnRleHRDb250ZW50PWNvdW50c1trXX0pfQogZnVuY3Rpb24gc2hvdygpe3ZhciBlPUVbaV07Y2FyZC5pbm5lckhUTUw9JzxkaXYgY2xhc3M9ImNkLXNtYWxsIj5FbWFpbCAnKyhpKzEpKycgb2YgJytFLmxlbmd0aCsnIMK3IFNjb3JlICcrc2NvcmUrJzwvZGl2PjxkaXYgY2xhc3M9ImNkLW1haWwiPiInK2UucysnIjwvZGl2PjxkaXYgY2xhc3M9ImNkLXJvdyIgc3R5bGU9ImdhcDo4cHg7bWFyZ2luLXRvcDo4cHgiPjxzcGFuIGNsYXNzPSJjZC1waWxsICcrKGUuYT8nY2QtcGlsbC1iYWQnOidjZC1waWxsLWdvb2QnKSsnIj5BY3R1YWxseTogJysoZS5hPydzcGFtJzonbm90IHNwYW0nKSsnPC9zcGFuPjxzcGFuIGNsYXNzPSJjZC1waWxsIj5Nb2RlbCBzYWlkOiAnKyhlLnA\/J3NwYW0gKG1vdmVkIHRvIHNwYW0gZm9sZGVyKSc6J25vdCBzcGFtIChrZXB0IGluIGluYm94KScpKyc8L3NwYW4+PC9kaXY+JzsKICBmYi5oaWRkZW49dHJ1ZTtueC5oaWRkZW49dHJ1ZTtidG5zLmZvckVhY2goZnVuY3Rpb24oYil7Yi5kaXNhYmxlZD1mYWxzZTtiLmNsYXNzTGlzdC5yZW1vdmUoImNkLXJpZ2h0IiwiY2Qtd3JvbmciKX0pfQogYnRucy5mb3JFYWNoKGZ1bmN0aW9uKGIpe2IuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7dmFyIGU9RVtpXSx0PXRydXRoKGUpLG9rPWIuZGF0YXNldC5rPT09dDtpZihvaylzY29yZSsrO2NvdW50c1t0XSsrO2dyaWQoKTsKICBidG5zLmZvckVhY2goZnVuY3Rpb24oeCl7eC5kaXNhYmxlZD10cnVlO2lmKHguZGF0YXNldC5rPT09dCl4LmNsYXNzTGlzdC5hZGQoImNkLXJpZ2h0Iik7ZWxzZSBpZih4PT09Yil4LmNsYXNzTGlzdC5hZGQoImNkLXdyb25nIil9KTsKICB2YXIgd2h5PShlLnA\/IlRoZSBtb2RlbCBzYWlkIDxiPnNwYW08L2I+IChwb3NpdGl2ZSkiOiJUaGUgbW9kZWwgc2FpZCA8Yj5ub3Qgc3BhbTwvYj4gKG5lZ2F0aXZlKSIpKyIsIGFuZCBpdCB3YXMgIisoZS5wPT09ZS5hPyI8Yj5yaWdodDwvYj4iOiI8Yj53cm9uZzwvYj4iKSsiLiBSaWdodCArIHBvc2l0aXZlID0gVHJ1ZSBQb3NpdGl2ZSwgd3JvbmcgKyBwb3NpdGl2ZSA9IEZhbHNlIFBvc2l0aXZlLCBhbmQgc28gb24uIjsKICBmYi5oaWRkZW49ZmFsc2U7ZmIuY2xhc3NOYW1lPSJjZC1mZWVkYmFjayAiKyhvaz8iY2Qtb2siOiJjZC1ubyIpO2ZiLmlubmVySFRNTD0iPGI+Iisob2s\/IkNvcnJlY3Q6ICI6Ikl0IGlzIGEgIikrTFt0XSsiLjwvYj4gIit3aHk7bnguaGlkZGVuPWZhbHNlO254LnRleHRDb250ZW50PWk8RS5sZW5ndGgtMT8iTmV4dCBlbWFpbCI6IlNlZSB0aGUgcmVzdWx0In0pfSk7CiB2YXIgZmluPWZhbHNlOwogbnguYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7aWYoZmluKXtmaW49ZmFsc2U7aT0wO3Njb3JlPTA7Y291bnRzPXtUUDowLEZQOjAsRk46MCxUTjowfTtncmlkKCk7c2hvdygpO3JldHVybn0KICBpZihpPEUubGVuZ3RoLTEpe2krKztzaG93KCk7cmV0dXJufQogIGZpbj10cnVlO2J0bnMuZm9yRWFjaChmdW5jdGlvbihiKXtiLmRpc2FibGVkPXRydWU7Yi5jbGFzc0xpc3QucmVtb3ZlKCJjZC1yaWdodCIsImNkLXdyb25nIil9KTtjYXJkLmlubmVySFRNTD0nPGRpdiBjbGFzcz0iY2QtbWFpbCI+WW91IHNvcnRlZCBhbGwgOCBlbWFpbHMuIFNjb3JlOiAnK3Njb3JlKycgLyA4PC9kaXY+PHAgY2xhc3M9ImNkLXNtYWxsIiBzdHlsZT0ibWFyZ2luOjZweCAwIDAiPlRoZSBncmlkIG5leHQgdG8gdGhpcyBib3ggaXMgbm93IGEgY29tcGxldGUgY29uZnVzaW9uIG1hdHJpeCBmb3IgdGhpcyB0aW55IHRlc3Q6IDIgdHJ1ZSBwb3NpdGl2ZXMsIDIgZmFsc2UgcG9zaXRpdmVzLCAyIGZhbHNlIG5lZ2F0aXZlcyBhbmQgMiB0cnVlIG5lZ2F0aXZlcy48L3A+JztmYi5oaWRkZW49dHJ1ZTtueC50ZXh0Q29udGVudD0iUGxheSBhZ2FpbiJ9KTsKIGdyaWQoKTtzaG93KCk7Cn0pOwoKLyogMi4gQWNjdXJhY3kgcGFyYWRveCAqLwokJCgiLmNkYWkuY2QtY20tcGFyYWRveCIpLmZvckVhY2goZnVuY3Rpb24oYm94KXsKIHZhciBzbD0kKCJpbnB1dCIsYm94KTsKIGZ1bmN0aW9uIGRyYXcoKXt2YXIgcmF0ZT0rc2wudmFsdWUvMTAwMCxuPTEwMDAwLHBvcz1NYXRoLnJvdW5kKG4qcmF0ZSksbmVnPW4tcG9zOwogICQoIi5jZC1wYXItcmF0ZSIsYm94KS50ZXh0Q29udGVudD0ocmF0ZSoxMDApLnRvRml4ZWQoMSkrIiUgKCIrcG9zKyIgb2YgMTAsMDAwKSI7CiAgdmFyIGxhenk9bWV0cmljcygwLDAscG9zLG5lZyksdHA9TWF0aC5yb3VuZChwb3MqMC44KSxmcD1NYXRoLnJvdW5kKG5lZyowLjA1KSxyZWFsPW1ldHJpY3ModHAsZnAscG9zLXRwLG5lZy1mcCk7CiAgZnVuY3Rpb24gZmlsbChzZWwsbSx0cHYsZnB2KXskKHNlbCsiIC5jZC1wYXItYWNjIixib3gpLnRleHRDb250ZW50PXBjdChtLmFjYyk7JChzZWwrIiAuY2QtcGFyLXJlYyIsYm94KS50ZXh0Q29udGVudD1wY3QobS5yZWMpOyQoc2VsKyIgLmNkLXBhci1jYXVnaHQiLGJveCkudGV4dENvbnRlbnQ9dHB2KyIgb2YgIitwb3M7JChzZWwrIiAuY2QtcGFyLWZhIixib3gpLnRleHRDb250ZW50PWZwdn0KICBmaWxsKCIuY2QtcGFyLWxhenkiLGxhenksMCwwKTtmaWxsKCIuY2QtcGFyLXJlYWwiLHJlYWwsdHAsZnApOwogICQoIi5jZC1wYXItbXNnIixib3gpLmlubmVySFRNTD1sYXp5LmFjYz5yZWFsLmFjYz8iVGhlIGxhenkgbW9kZWwgaGFzIDxiPmhpZ2hlciBhY2N1cmFjeTwvYj4gKCIrcGN0KGxhenkuYWNjKSsiIHZzICIrcGN0KHJlYWwuYWNjKSsiKSBhbmQgY2F0Y2hlcyA8Yj56ZXJvPC9iPiBmcmF1ZC4gQWNjdXJhY3kgaXMgcmV3YXJkaW5nIGl0IGZvciBwcmVkaWN0aW5nIHRoZSBtYWpvcml0eSBjbGFzcy4iOiJXaXRoIHRoaXMgbWFueSBmcmF1ZCBjYXNlcywgdGhlIGxhenkgbW9kZWwncyBhY2N1cmFjeSBmaW5hbGx5IGRyb3BzIGJlbG93IHRoZSByZWFsIG1vZGVsLiBPbiByYXJlLWV2ZW50IHByb2JsZW1zLCB0aG91Z2gsIGFjY3VyYWN5IGFsb25lIHdpbGwgbWlzbGVhZCB5b3UuIn0KIHNsLmFkZEV2ZW50TGlzdGVuZXIoImlucHV0IixkcmF3KTtkcmF3KCk7Cn0pOwoKLyogMy4gTWV0cmljcyBjYWxjdWxhdG9yICovCiQkKCIuY2RhaS5jZC1jbS1jYWxjIikuZm9yRWFjaChmdW5jdGlvbihib3gpewogdmFyIGlucz17fTtbInRwIiwiZnAiLCJmbiIsInRuIl0uZm9yRWFjaChmdW5jdGlvbihrKXtpbnNba109JCgiLmNkLWluLSIrayxib3gpfSk7CiB2YXIgUD17c3BhbTpbNCwyLDEsNV0sZnJhdWQ6WzgsNDUsMiw5NDVdLHBlcmZlY3Q6WzUwLDAsMCw1MF0sYWxsOlsxMCw5MCwwLDBdfTsKIGZ1bmN0aW9uIHYoayl7dmFyIHg9cGFyc2VJbnQoaW5zW2tdLnZhbHVlLDEwKTtyZXR1cm4gaXNOYU4oeCl8fHg8MD8wOnh9CiBmdW5jdGlvbiBkcmF3KCl7dmFyIHRwPXYoInRwIiksZnA9digiZnAiKSxmbj12KCJmbiIpLHRuPXYoInRuIiksbT1tZXRyaWNzKHRwLGZwLGZuLHRuKSxvdXQ9JCgiLmNkLWNhbGMtb3V0Iixib3gpOwogIHZhciByb3dzPVtbIkFjY3VyYWN5IiwiKFRQICsgVE4pIC8gVG90YWwiLCIoIit0cCsiICsgIit0bisiKSAvICIrbS5uLG0uYWNjLCJTaGFyZSBvZiBhbGwgcHJlZGljdGlvbnMgdGhhdCB3ZXJlIHJpZ2h0Il0sCiAgIFsiUHJlY2lzaW9uIiwiVFAgLyAoVFAgKyBGUCkiLHRwKyIgLyAoIit0cCsiICsgIitmcCsiKSIsbS5wcmVjLCJPZiBldmVyeXRoaW5nIGZsYWdnZWQgcG9zaXRpdmUsIGhvdyBtdWNoIHJlYWxseSB3YXMiXSwKICAgWyJSZWNhbGwgKHNlbnNpdGl2aXR5LCBUUFIpIiwiVFAgLyAoVFAgKyBGTikiLHRwKyIgLyAoIit0cCsiICsgIitmbisiKSIsbS5yZWMsIk9mIGFsbCByZWFsIHBvc2l0aXZlcywgaG93IG1hbnkgd2VyZSBjYXVnaHQiXSwKICAgWyJTcGVjaWZpY2l0eSAoVE5SKSIsIlROIC8gKFROICsgRlApIix0bisiIC8gKCIrdG4rIiArICIrZnArIikiLG0uc3BlYywiT2YgYWxsIHJlYWwgbmVnYXRpdmVzLCBob3cgbWFueSB3ZXJlIGNvcnJlY3RseSBjbGVhcmVkIl0sCiAgIFsiRmFsc2UgcG9zaXRpdmUgcmF0ZSIsIkZQIC8gKEZQICsgVE4pIixmcCsiIC8gKCIrZnArIiArICIrdG4rIikiLG0uZnByLCJSZWFsIG5lZ2F0aXZlcyB3cm9uZ2x5IGZsYWdnZWQgKDEg4oiSIHNwZWNpZmljaXR5KSJdLAogICBbIkYxIHNjb3JlIiwiMiDDlyBQIMOXIFIgLyAoUCArIFIpIiwiMiDDlyAiK2YzKG0ucHJlYykrIiDDlyAiK2YzKG0ucmVjKSsiIC8gKCIrZjMobS5wcmVjKSsiICsgIitmMyhtLnJlYykrIikiLG0uZjEsIkJhbGFuY2Ugb2YgcHJlY2lzaW9uIGFuZCByZWNhbGwiXV07CiAgb3V0LmlubmVySFRNTD0nPGRpdiBjbGFzcz0iY2QtdGJsLXdyYXAiPjx0YWJsZSBjbGFzcz0iY2QtdGJsIGNkLW10YWIiPjx0aGVhZD48dHI+PHRoPk1ldHJpYzwvdGg+PHRoPkZvcm11bGE8L3RoPjx0aD5XaXRoIHlvdXIgbnVtYmVyczwvdGg+PHRoPlJlc3VsdDwvdGg+PC90cj48L3RoZWFkPjx0Ym9keT4nK3Jvd3MubWFwKGZ1bmN0aW9uKHIpe3JldHVybiAnPHRyPjx0ZD48Yj4nK3JbMF0rJzwvYj48ZGl2IGNsYXNzPSJjZC1zbWFsbCI+JytyWzRdKyc8L2Rpdj48L3RkPjx0ZD48Y29kZT4nK3JbMV0rJzwvY29kZT48L3RkPjx0ZD48Y29kZT4nK3JbMl0rJzwvY29kZT48L3RkPjx0ZCBjbGFzcz0iY2QtcmVzLXYiPicrKGlzRmluaXRlKHJbM10pP2YzKHJbM10pOic8c3BhbiB0aXRsZT0iZGl2aXNpb24gYnkgemVybyI+dW5kZWZpbmVkPC9zcGFuPicpKyc8L3RkPjwvdHI+J30pLmpvaW4oIiIpKyc8L3Rib2R5PjwvdGFibGU+PC9kaXY+JzsKICAkKCIuY2QtY2FsYy1lcnIiLGJveCkuaW5uZXJIVE1MPSJUeXBlIEkgZXJyb3JzIChmYWxzZSBwb3NpdGl2ZXMpOiA8Yj4iK2ZwKyI8L2I+IMK3IFR5cGUgSUkgZXJyb3JzIChmYWxzZSBuZWdhdGl2ZXMpOiA8Yj4iK2ZuKyI8L2I+IMK3IFRvdGFsOiA8Yj4iK20ubisiPC9iPiJ9CiBPYmplY3Qua2V5cyhpbnMpLmZvckVhY2goZnVuY3Rpb24oayl7aW5zW2tdLmFkZEV2ZW50TGlzdGVuZXIoImlucHV0IixkcmF3KX0pOwogJCQoIi5jZC1wcmVzZXQiLGJveCkuZm9yRWFjaChmdW5jdGlvbihiKXtiLmFkZEV2ZW50TGlzdGVuZXIoImNsaWNrIixmdW5jdGlvbigpe3ZhciBwPVBbYi5kYXRhc2V0LnBdO1sidHAiLCJmcCIsImZuIiwidG4iXS5mb3JFYWNoKGZ1bmN0aW9uKGssaSl7aW5zW2tdLnZhbHVlPXBbaV19KTtkcmF3KCk7JCQoIi5jZC1wcmVzZXQiLGJveCkuZm9yRWFjaChmdW5jdGlvbih4KXt4LnNldEF0dHJpYnV0ZSgiYXJpYS1wcmVzc2VkIix4PT09Yj8idHJ1ZSI6ImZhbHNlIil9KX0pfSk7CiBkcmF3KCk7Cn0pOwoKLyogNC4gRjEgZXhwbG9yZXIgKi8KJCQoIi5jZGFpLmNkLWNtLWYxIikuZm9yRWFjaChmdW5jdGlvbihib3gpewogdmFyIHBzPSQoIi5jZC1mMS1wIixib3gpLHJzPSQoIi5jZC1mMS1yIixib3gpLGJzPSQoIi5jZC1mMS1iIixib3gpOwogZnVuY3Rpb24gZHJhdygpe3ZhciBwPStwcy52YWx1ZS8xMDAscj0rcnMudmFsdWUvMTAwLGI9K2JzLnZhbHVlLG1lYW49KHArcikvMixmMT1wK3I\/MipwKnIvKHArcik6MCxmYj1wK3I\/KDErYipiKSpwKnIvKGIqYipwK3IpOjA7CiAgJCgiLmNkLWYxLXB2Iixib3gpLnRleHRDb250ZW50PXAudG9GaXhlZCgyKTskKCIuY2QtZjEtcnYiLGJveCkudGV4dENvbnRlbnQ9ci50b0ZpeGVkKDIpOyQoIi5jZC1mMS1idiIsYm94KS50ZXh0Q29udGVudD1iOwogIFtbIm1lYW4iLG1lYW5dLFsiZjEiLGYxXSxbImZiIixmYl1dLmZvckVhY2goZnVuY3Rpb24oeCl7JCgiLmNkLWYxLSIreFswXSsiLXYiLGJveCkudGV4dENvbnRlbnQ9eFsxXS50b0ZpeGVkKDMpOyQoIi5jZC1mMS0iK3hbMF0rIi1iYXIiLGJveCkuc3R5bGUud2lkdGg9KHhbMV0qMTAwKSsiJSJ9KTsKICAkKCIuY2QtZjEtZmItbCIsYm94KS50ZXh0Q29udGVudD0iRiIrYisiIHNjb3JlIjsKICAkKCIuY2QtZjEtbXNnIixib3gpLnRleHRDb250ZW50PU1hdGguYWJzKHAtcik+MC41PyJQcmVjaXNpb24gYW5kIHJlY2FsbCBhcmUgZmFyIGFwYXJ0LCBzbyBGMSBzdGF5cyBsb3cgZXZlbiB0aG91Z2ggdGhlIHNpbXBsZSBhdmVyYWdlIGxvb2tzIGZpbmUuIFRoZSBoYXJtb25pYyBtZWFuIGlzIHB1bGxlZCB0b3dhcmRzIHRoZSBzbWFsbGVyIHZhbHVlLiI6TWF0aC5hYnMocC1yKTwwLjA1PyJQcmVjaXNpb24gYW5kIHJlY2FsbCBhcmUgY2xvc2UsIHNvIEYxIGlzIGFsbW9zdCBlcXVhbCB0byB0aGUgc2ltcGxlIGF2ZXJhZ2UuIjpiPjE\/IldpdGggYmV0YSBhYm92ZSAxLCB0aGUgRi1iZXRhIHNjb3JlIGxlYW5zIHRvd2FyZHMgcmVjYWxsLCBzbyBhIG1vZGVsIHRoYXQgbWlzc2VzIGZld2VyIHBvc2l0aXZlcyBzY29yZXMgaGlnaGVyLiI6YjwxPyJXaXRoIGJldGEgYmVsb3cgMSwgdGhlIEYtYmV0YSBzY29yZSBsZWFucyB0b3dhcmRzIHByZWNpc2lvbi4iOiJNb3ZlIHRoZSBzbGlkZXJzIGFwYXJ0IGFuZCB3YXRjaCBGMSBkcm9wIGZhc3RlciB0aGFuIHRoZSBzaW1wbGUgYXZlcmFnZS4ifQogW3BzLHJzLGJzXS5mb3JFYWNoKGZ1bmN0aW9uKHMpe3MuYWRkRXZlbnRMaXN0ZW5lcigiaW5wdXQiLGRyYXcpfSk7ZHJhdygpOwp9KTsKCi8qIDUuIFRocmVzaG9sZCwgUk9DIGFuZCBQUiBleHBsb3JlciAqLwokJCgiLmNkYWkuY2QtY20tdGhyIikuZm9yRWFjaChmdW5jdGlvbihib3gpewogdmFyIGN2cz0kJCgiY2FudmFzIixib3gpLFMxLFMyLFMzLGRhdGE9W10sdGhyPTAuNSxzZXA9MS42OwogZnVuY3Rpb24gcSh1KXtyZXR1cm4gTWF0aC5sb2codS8oMS11KSkvMS43MDJ9ZnVuY3Rpb24gZ2VuKCl7ZGF0YT1bXTtmb3IodmFyIGk9MDtpPDYwO2krKyl7dmFyIHBvcz1pPDIwLGs9cG9zP2k6aS0yMCxuPXBvcz8yMDo0MCx6PShwb3M\/c2VwLzI6LXNlcC8yKStxKChrKzAuNSkvbik7ZGF0YS5wdXNoKHt5OnBvcz8xOjAsczoxLygxK01hdGguZXhwKC16KjEuMykpfSl9fQogZnVuY3Rpb24gY291bnRzKHQpe3ZhciBjPXt0cDowLGZwOjAsZm46MCx0bjowfTtkYXRhLmZvckVhY2goZnVuY3Rpb24oZCl7dmFyIHA9ZC5zPj10O2lmKHAmJmQueSljLnRwKys7ZWxzZSBpZihwKWMuZnArKztlbHNlIGlmKGQueSljLmZuKys7ZWxzZSBjLnRuKyt9KTtyZXR1cm4gY30KIGZ1bmN0aW9uIGN1cnZlcygpe3ZhciB0cz1kYXRhLm1hcChmdW5jdGlvbihkKXtyZXR1cm4gZC5zfSkuY29uY2F0KFsxLjAxLC0wLjAxXSkuc29ydChmdW5jdGlvbihhLGIpe3JldHVybiBiLWF9KSxyb2M9W10scHI9W107dHMuZm9yRWFjaChmdW5jdGlvbih0KXt2YXIgYz1jb3VudHModCksdHByPWMudHAvKGMudHArYy5mbiksZnByPWMuZnAvKGMuZnArYy50bikscHJlYz1jLnRwK2MuZnA\/Yy50cC8oYy50cCtjLmZwKToxO3JvYy5wdXNoKFtmcHIsdHByXSk7cHIucHVzaChbdHByLHByZWNdKX0pOwogIHZhciBhdWM9MDtmb3IodmFyIGk9MTtpPHJvYy5sZW5ndGg7aSsrKWF1Yys9KHJvY1tpXVswXS1yb2NbaS0xXVswXSkqKHJvY1tpXVsxXStyb2NbaS0xXVsxXSkvMjtyZXR1cm4ge3JvYzpyb2MscHI6cHIsYXVjOmF1Y319CiBmdW5jdGlvbiBheGVzKFMsYyxvLHhsLHlsKXt2YXIgQz1jb2woYm94KTtjLnN0cm9rZVN0eWxlPUMubGluZTtjLmxpbmVXaWR0aD0xO2Muc3Ryb2tlUmVjdChvLkwsby5ULFMudy1vLkwtby5SLFMuaC1vLlQtby5CKTtjLmZpbGxTdHlsZT1DLm11dDtjLmZvbnQ9IjExcHggSW50ZXIsc2Fucy1zZXJpZiI7Yy5maWxsVGV4dCh4bCxvLkwrKFMudy1vLkwtby5SKS8yLWMubWVhc3VyZVRleHQoeGwpLndpZHRoLzIsUy5oLTQpO2Muc2F2ZSgpO2MudHJhbnNsYXRlKDExLG8uVCsoUy5oLW8uVC1vLkIpLzIrYy5tZWFzdXJlVGV4dCh5bCkud2lkdGgvMik7Yy5yb3RhdGUoLU1hdGguUEkvMik7Yy5maWxsVGV4dCh5bCwwLDApO2MucmVzdG9yZSgpO2MuZmlsbFRleHQoIjAiLG8uTC0xMCxTLmgtby5CKzEyKTtjLmZpbGxUZXh0KCIxIixTLnctby5SLTQsUy5oLW8uQisxMik7Yy5maWxsVGV4dCgiMSIsby5MLTEwLG8uVCs4KX0KIGZ1bmN0aW9uIGRyYXcoKXtpZighUzEpe1MxPWN2MihjdnNbMF0sMTUwKTtTMj1jdjIoY3ZzWzFdLDIzMCk7UzM9Y3YyKGN2c1syXSwyMzApfXZhciBDPWNvbChib3gpLGM9UzEuYyxMPTcwLFI9MTQsY250PWNvdW50cyh0aHIpLG09bWV0cmljcyhjbnQudHAsY250LmZwLGNudC5mbixjbnQudG4pLGN1PWN1cnZlcygpOwogIGZ1bmN0aW9uIHN4KHYpe3JldHVybiBMK3YqKFMxLnctTC1SKX0KICBjLmNsZWFyUmVjdCgwLDAsUzEudyxTMS5oKTtjLmZpbGxTdHlsZT1DLmJyO2MuZ2xvYmFsQWxwaGE9LjA4O2MuZmlsbFJlY3Qoc3godGhyKSw4LFMxLnctUi1zeCh0aHIpLDExMCk7Yy5nbG9iYWxBbHBoYT0xOwogIGMuZm9udD0iMTFweCBJbnRlcixzYW5zLXNlcmlmIjtjLmZpbGxTdHlsZT1DLm11dDtjLmZpbGxUZXh0KCJzcGFtIiw4LDQ0KTtjLmZpbGxUZXh0KCJub3Qgc3BhbSIsOCw5Mik7Yy5maWxsVGV4dCgibW9kZWwgc2NvcmUg4oaSIixTMS53LzItMzAsMTQ2KTtjLmZpbGxUZXh0KCIwIixMLTMsMTQ2KTtjLmZpbGxUZXh0KCIxIixTMS53LVItNCwxNDYpOwogIGMuZmlsbFN0eWxlPUMuYnI7Yy5maWxsVGV4dCgicHJlZGljdGVkIHNwYW0g4oaSIixNYXRoLm1pbihzeCh0aHIpKzQsUzEudy1SLTk2KSwyMCk7CiAgdmFyIHI9cm5nKDcpO2RhdGEuZm9yRWFjaChmdW5jdGlvbihkKXt2YXIgeT0oZC55PzQwOjg4KSsocigpLTAuNSkqMTg7Yy5maWxsU3R5bGU9ZC55P0MuYmFkOkMuZ29vZDtjLmJlZ2luUGF0aCgpO2MuYXJjKHN4KGQucykseSw1LDAsNyk7Yy5maWxsKCl9KTsKICBjLnN0cm9rZVN0eWxlPUMuaW5rO2MubGluZVdpZHRoPTIuNTtjLmJlZ2luUGF0aCgpO2MubW92ZVRvKHN4KHRociksOCk7Yy5saW5lVG8oc3godGhyKSwxMTgpO2Muc3Ryb2tlKCk7Yy5saW5lV2lkdGg9MTsKICBbInRwIiwiZnAiLCJmbiIsInRuIl0uZm9yRWFjaChmdW5jdGlvbihrKXskKCIuY2QtdC0iK2ssYm94KS50ZXh0Q29udGVudD1jbnRba119KTsKICAkKCIuY2QtdC1wcmVjIixib3gpLnRleHRDb250ZW50PXBjdChtLnByZWMpOyQoIi5jZC10LXJlYyIsYm94KS50ZXh0Q29udGVudD1wY3QobS5yZWMpOyQoIi5jZC10LWZwciIsYm94KS50ZXh0Q29udGVudD1wY3QobS5mcHIpOyQoIi5jZC10LWYxIixib3gpLnRleHRDb250ZW50PWYzKG0uZjEpOyQoIi5jZC10LWFjYyIsYm94KS50ZXh0Q29udGVudD1wY3QobS5hY2MpOyQoIi5jZC10LXRodiIsYm94KS50ZXh0Q29udGVudD10aHIudG9GaXhlZCgyKTsKICAvLyBST0MKICB2YXIgbz17TDoyNixSOjEwLFQ6MTAsQjoyMH0sZD1TMi5jLFc9UzIudy1vLkwtby5SLEg9UzIuaC1vLlQtby5CO2QuY2xlYXJSZWN0KDAsMCxTMi53LFMyLmgpO2F4ZXMoUzIsZCxvLCJmYWxzZSBwb3NpdGl2ZSByYXRlIiwidHJ1ZSBwb3NpdGl2ZSByYXRlIChyZWNhbGwpIik7CiAgZC5zZXRMaW5lRGFzaChbNCw0XSk7ZC5zdHJva2VTdHlsZT1DLm11dDtkLmJlZ2luUGF0aCgpO2QubW92ZVRvKG8uTCxvLlQrSCk7ZC5saW5lVG8oby5MK1csby5UKTtkLnN0cm9rZSgpO2Quc2V0TGluZURhc2goW10pOwogIGQuc3Ryb2tlU3R5bGU9Qy5icjtkLmxpbmVXaWR0aD0yLjU7ZC5iZWdpblBhdGgoKTtjdS5yb2MuZm9yRWFjaChmdW5jdGlvbihwLGkpe3ZhciB4PW8uTCtwWzBdKlcseT1vLlQrSC1wWzFdKkg7aT9kLmxpbmVUbyh4LHkpOmQubW92ZVRvKHgseSl9KTtkLnN0cm9rZSgpO2QubGluZVdpZHRoPTE7CiAgZC5maWxsU3R5bGU9Qy5iYWQ7ZC5iZWdpblBhdGgoKTtkLmFyYyhvLkwrbS5mcHIqVyxvLlQrSC1tLnJlYypILDYsMCw3KTtkLmZpbGwoKTtkLmZpbGxTdHlsZT1DLmluaztkLmZvbnQ9ImJvbGQgMTJweCBJbnRlcixzYW5zLXNlcmlmIjtkLmZpbGxUZXh0KCJBVUMgPSAiK2N1LmF1Yy50b0ZpeGVkKDMpLG8uTCtXLTkyLG8uVCtILTEwKTsKICAvLyBQUgogIHZhciBlPVMzLmM7ZS5jbGVhclJlY3QoMCwwLFMzLncsUzMuaCk7YXhlcyhTMyxlLG8sInJlY2FsbCIsInByZWNpc2lvbiIpO2Uuc3Ryb2tlU3R5bGU9Qy5nb29kO2UubGluZVdpZHRoPTIuNTtlLmJlZ2luUGF0aCgpO3ZhciBmaXJzdD10cnVlO2N1LnByLmZvckVhY2goZnVuY3Rpb24ocCl7aWYoIWlzRmluaXRlKHBbMF0pKXJldHVybjt2YXIgeD1vLkwrcFswXSpXLHk9by5UK0gtcFsxXSpIO2ZpcnN0P2UubW92ZVRvKHgseSk6ZS5saW5lVG8oeCx5KTtmaXJzdD1mYWxzZX0pO2Uuc3Ryb2tlKCk7ZS5saW5lV2lkdGg9MTsKICBpZihpc0Zpbml0ZShtLnByZWMpKXtlLmZpbGxTdHlsZT1DLmJhZDtlLmJlZ2luUGF0aCgpO2UuYXJjKG8uTCttLnJlYypXLG8uVCtILW0ucHJlYypILDYsMCw3KTtlLmZpbGwoKX0KICAkKCIuY2QtdC1hdWMiLGJveCkudGV4dENvbnRlbnQ9Y3UuYXVjLnRvRml4ZWQoMyk7CiAgJCgiLmNkLXQtbXNnIixib3gpLnRleHRDb250ZW50PXNlcDwwLjQ\/IlRoZSB0d28gZ3JvdXBzIG92ZXJsYXAgYWxtb3N0IGNvbXBsZXRlbHksIHNvIG5vIHRocmVzaG9sZCBzZXBhcmF0ZXMgdGhlbS4gVGhlIFJPQyBjdXJ2ZSBodWdzIHRoZSBkaWFnb25hbCBhbmQgQVVDIGlzIG5lYXIgMC41OiB0aGUgbW9kZWwgaXMgY2xvc2UgdG8gcmFuZG9tIGd1ZXNzaW5nLiI6dGhyPjAuNzU\/IkhpZ2ggdGhyZXNob2xkOiBvbmx5IHZlcnkgY29uZmlkZW50IHByZWRpY3Rpb25zIGFyZSBmbGFnZ2VkLiBQcmVjaXNpb24gaXMgaGlnaCwgYnV0IHJlY2FsbCBkcm9wcyBiZWNhdXNlIG1hbnkgcmVhbCBzcGFtIGVtYWlscyBzbGlwIHVuZGVyIHRoZSBsaW5lLiI6dGhyPDAuMz8iTG93IHRocmVzaG9sZDogYWxtb3N0IGV2ZXJ5dGhpbmcgc3VzcGljaW91cyBpcyBmbGFnZ2VkLiBSZWNhbGwgaXMgaGlnaCwgYnV0IHByZWNpc2lvbiBmYWxscyBiZWNhdXNlIGdlbnVpbmUgZW1haWxzIGdldCBjYXVnaHQgdG9vLiI6IkRyYWcgdGhlIHRocmVzaG9sZCBhbmQgd2F0Y2ggb25lIHJlZCBkb3QgbW92ZSBhbG9uZyB0aGUgUk9DIGFuZCBwcmVjaXNpb24tcmVjYWxsIGN1cnZlcy4gVGhlbiBjaGFuZ2UgdGhlIG1vZGVsIHF1YWxpdHkuIn0KICQoIi5jZC10LXRoIixib3gpLmFkZEV2ZW50TGlzdGVuZXIoImlucHV0IixmdW5jdGlvbihlKXt0aHI9K2UudGFyZ2V0LnZhbHVlO2RyYXcoKX0pOwogJCgiLmNkLXQtc2VwIixib3gpLmFkZEV2ZW50TGlzdGVuZXIoImlucHV0IixmdW5jdGlvbihlKXtzZXA9K2UudGFyZ2V0LnZhbHVlOyQoIi5jZC10LXNlcHYiLGJveCkudGV4dENvbnRlbnQ9c2VwPDAuND8icmFuZG9tIjpzZXA8MS4yPyJ3ZWFrIjpzZXA8Mi40PyJnb29kIjoiZXhjZWxsZW50IjtnZW4oKTtkcmF3KCl9KTsKIHdpbmRvdy5hZGRFdmVudExpc3RlbmVyKCJyZXNpemUiLGZ1bmN0aW9uKCl7UzE9bnVsbDtkcmF3KCl9KTtnZW4oKTtkcmF3KCk7Cn0pOwoKLyogNi4gTXVsdGktY2xhc3MgKi8KJCQoIi5jZGFpLmNkLWNtLW11bHRpIikuZm9yRWFjaChmdW5jdGlvbihib3gpewogdmFyIE49WyJCaWxsaW5nIiwiVGVjaG5pY2FsIiwiQWNjb3VudCJdLE09W1s4LDEsMV0sWzIsOSwxXSxbMSwyLDVdXSxzZWw9MDsKIGZ1bmN0aW9uIHN0YXRzKGspe3ZhciB0cD1NW2tdW2tdLGZuPTAsZnA9MCx0b3Q9MDtmb3IodmFyIGk9MDtpPDM7aSsrKWZvcih2YXIgaj0wO2o8MztqKyspe3RvdCs9TVtpXVtqXTtpZihpPT09ayYmaiE9PWspZm4rPU1baV1bal07aWYoaj09PWsmJmkhPT1rKWZwKz1NW2ldW2pdfXZhciBwPXRwLyh0cCtmcCkscj10cC8odHArZm4pO3JldHVybiB7dHA6dHAsZnA6ZnAsZm46Zm4sdG46dG90LXRwLWZwLWZuLHA6cCxyOnIsZjoyKnAqci8ocCtyKSxzdXA6dHArZm4sdG90OnRvdH19CiBmdW5jdGlvbiBkcmF3KCl7dmFyIGc9JCgiLmNkLW1jLWdyaWQiLGJveCksaD0nPGRpdiBjbGFzcz0iY2QtbWMtYyBjZC1tYy1oIj48L2Rpdj4nK04ubWFwKGZ1bmN0aW9uKG4pe3JldHVybiAnPGRpdiBjbGFzcz0iY2QtbWMtYyBjZC1tYy1oIj5QcmVkOiAnK24rJzwvZGl2Pid9KS5qb2luKCIiKTsKICBmb3IodmFyIGk9MDtpPDM7aSsrKXtoKz0nPGRpdiBjbGFzcz0iY2QtbWMtYyBjZC1tYy1oIj5BY3R1YWw6ICcrTltpXSsnPC9kaXY+Jztmb3IodmFyIGo9MDtqPDM7aisrKXt2YXIgY2xzPWk9PT1zZWwmJmo9PT1zZWw\/ImNkLWstdHAiOmk9PT1zZWw\/ImNkLWstZm4iOmo9PT1zZWw\/ImNkLWstZnAiOiJjZC1rLXRuIjtoKz0nPGRpdiBjbGFzcz0iY2QtbWMtYyAnK2NscysnIj4nK01baV1bal0rJzwvZGl2Pid9fWcuaW5uZXJIVE1MPWg7CiAgdmFyIHM9c3RhdHMoc2VsKTskKCIuY2QtbWMtZGV0YWlsIixib3gpLmlubmVySFRNTD0nPGI+JytOW3NlbF0rJzwvYj4gYXMgdGhlIHBvc2l0aXZlIGNsYXNzOiBUUCA9ICcrcy50cCsnLCBGTiA9ICcrcy5mbisnIChyZXN0IG9mIGl0cyByb3cpLCBGUCA9ICcrcy5mcCsnIChyZXN0IG9mIGl0cyBjb2x1bW4pLCBUTiA9ICcrcy50bisnLjxicj5QcmVjaXNpb24gPSAnK3MudHArJyAvICgnK3MudHArJyArICcrcy5mcCsnKSA9IDxiPicrZjMocy5wKSsnPC9iPiDCtyBSZWNhbGwgPSAnK3MudHArJyAvICgnK3MudHArJyArICcrcy5mbisnKSA9IDxiPicrZjMocy5yKSsnPC9iPiDCtyBGMSA9IDxiPicrZjMocy5mKSsnPC9iPic7CiAgdmFyIGFsbD1bMCwxLDJdLm1hcChzdGF0cyksdG90PWFsbFswXS50b3QsbWFjcm89YWxsLnJlZHVjZShmdW5jdGlvbihhLHgpe3JldHVybiBhK3guZn0sMCkvMyx3ZWlnaHRlZD1hbGwucmVkdWNlKGZ1bmN0aW9uKGEseCl7cmV0dXJuIGEreC5mKnguc3VwfSwwKS90b3QsbWljcm89YWxsLnJlZHVjZShmdW5jdGlvbihhLHgpe3JldHVybiBhK3gudHB9LDApL3RvdDsKICAkKCIuY2QtbWMtYXZnIixib3gpLmlubmVySFRNTD0nPGRpdiBjbGFzcz0iY2QtY2FyZCI+PGRpdiBjbGFzcz0iY2Qtc21hbGwiPk1hY3JvIEYxIChzaW1wbGUgYXZlcmFnZSBvZiB0aGUgMyBjbGFzc2VzKTwvZGl2PjxkaXYgY2xhc3M9ImNkLW1ldGVyIj4nK2YzKG1hY3JvKSsnPC9kaXY+PC9kaXY+PGRpdiBjbGFzcz0iY2QtY2FyZCI+PGRpdiBjbGFzcz0iY2Qtc21hbGwiPldlaWdodGVkIEYxIChhdmVyYWdlIHdlaWdodGVkIGJ5IGNsYXNzIHNpemUpPC9kaXY+PGRpdiBjbGFzcz0iY2QtbWV0ZXIiPicrZjMod2VpZ2h0ZWQpKyc8L2Rpdj48L2Rpdj48ZGl2IGNsYXNzPSJjZC1jYXJkIj48ZGl2IGNsYXNzPSJjZC1zbWFsbCI+TWljcm8gRjEgKHBvb2wgYWxsIFRQLCBGUCwgRk47IGVxdWFscyBhY2N1cmFjeSBoZXJlKTwvZGl2PjxkaXYgY2xhc3M9ImNkLW1ldGVyIj4nK2YzKG1pY3JvKSsnPC9kaXY+PC9kaXY+JzsKICAkJCgiLmNkLWNoaXAiLGJveCkuZm9yRWFjaChmdW5jdGlvbihjLGspe2Muc2V0QXR0cmlidXRlKCJhcmlhLXByZXNzZWQiLGs9PT1zZWw\/InRydWUiOiJmYWxzZSIpfSl9CiAkJCgiLmNkLWNoaXAiLGJveCkuZm9yRWFjaChmdW5jdGlvbihjLGspe2MuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7c2VsPWs7ZHJhdygpfSl9KTtkcmF3KCk7Cn0pOwoKLyogNy4gV2hpY2ggbWV0cmljICovCiQkKCIuY2RhaS5jZC1jbS13aGljaCIpLmZvckVhY2goZnVuY3Rpb24oYm94KXsKIHZhciBTPVsKICB7dDoiQSBjYW5jZXIgc2NyZWVuaW5nIHRlc3QuIFBvc2l0aXZlID0gcG9zc2libGUgY2FuY2VyLCBzZW50IGZvciBhIGJpb3BzeS4iLGE6IlJlY2FsbCIsZToiTWlzc2luZyBhIHJlYWwgY2FuY2VyIChmYWxzZSBuZWdhdGl2ZSkgY2FuIGNvc3QgYSBsaWZlLCB3aGlsZSBhIGZhbHNlIGFsYXJtIG9ubHkgbWVhbnMgYW4gZXh0cmEgdGVzdC4gTWF4aW1pc2UgcmVjYWxsLCB0aGVuIGtlZXAgcHJlY2lzaW9uIHJlYXNvbmFibGUuIn0sCiAge3Q6IkEgc3BhbSBmaWx0ZXIgZm9yIGpvYiBzZWVrZXJzJyBpbmJveGVzLiBQb3NpdGl2ZSA9IG1vdmVkIHRvIHNwYW0uIixhOiJQcmVjaXNpb24iLGU6IkEgZ2VudWluZSBpbnRlcnZpZXcgZW1haWwgbG9zdCBpbiBzcGFtIChmYWxzZSBwb3NpdGl2ZSkgaXMgZmFyIHdvcnNlIHRoYW4gb25lIHNwYW0gZW1haWwgcmVhY2hpbmcgdGhlIGluYm94LiBQcmVjaXNpb24gbWF0dGVycyBtb3N0LiJ9LAogIHt0OiJDcmVkaXQgY2FyZCBmcmF1ZCBkZXRlY3Rpb24sIHdoZXJlIDAuMiUgb2YgdHJhbnNhY3Rpb25zIGFyZSBmcmF1ZC4iLGE6IkYxIixlOiJUaGUgZGF0YSBpcyBleHRyZW1lbHkgaW1iYWxhbmNlZCwgc28gYWNjdXJhY3kgaXMgdXNlbGVzcyAoOTkuOCUgYnkgcHJlZGljdGluZyAnbm90IGZyYXVkJykuIFlvdSBuZWVkIHRvIGNhdGNoIGZyYXVkIHdpdGhvdXQgYmxvY2tpbmcgdG9vIG1hbnkgZ2VudWluZSBwYXltZW50cywgc28gYmFsYW5jZSBib3RoIHdpdGggRjEsIGFuZCBsb29rIGF0IHRoZSBwcmVjaXNpb24tcmVjYWxsIGN1cnZlLiJ9LAogIHt0OiJDbGFzc2lmeWluZyBwaG90b3MgYXMgY2F0IG9yIGRvZywgd2l0aCA1LDAwMCBvZiBlYWNoIGFuZCBlcXVhbCBjb3N0IGZvciBlaXRoZXIgbWlzdGFrZS4iLGE6IkFjY3VyYWN5IixlOiJUaGUgY2xhc3NlcyBhcmUgYmFsYW5jZWQgYW5kIGJvdGggZXJyb3JzIGNvc3QgdGhlIHNhbWUsIHNvIHBsYWluIGFjY3VyYWN5IGlzIGEgZmFpciBhbmQgZWFzeS10by1leHBsYWluIHN1bW1hcnkuIn0sCiAge3Q6IkEgWW91VHViZS1zdHlsZSByZWNvbW1lbmRlciBzaG93cyAxMCB2aWRlb3MuIFBvc2l0aXZlID0gdGhlIHVzZXIgd291bGQgbGlrZSBpdC4iLGE6IlByZWNpc2lvbiIsZToiVXNlcnMgb25seSBzZWUgYSBmZXcgcmVzdWx0cywgc28gZXZlcnkgc2xvdCBzaG93biBzaG91bGQgYmUgcmVsZXZhbnQuIE1pc3Npbmcgc29tZSBnb29kIHZpZGVvcyAobG93IHJlY2FsbCkgbWF0dGVycyBtdWNoIGxlc3MuIn0sCiAge3Q6IkFpcnBvcnQgc2VjdXJpdHkgZmxhZ3MgYmFncyBmb3IgYSBtYW51YWwgY2hlY2suIFBvc2l0aXZlID0gY29udGFpbnMgYSBkYW5nZXJvdXMgaXRlbS4iLGE6IlJlY2FsbCIsZToiQSBkYW5nZXJvdXMgaXRlbSB0aGF0IHNsaXBzIHRocm91Z2ggaXMgdW5hY2NlcHRhYmxlLCBzbyB0aGUgc3lzdGVtIGZsYWdzIGdlbmVyb3VzbHkgYW5kIGFjY2VwdHMgbWFueSBmYWxzZSBhbGFybXMuIn1dOwogdmFyIGk9MCxzY29yZT0wLGJvZHk9JCgiLmNkLXdoLWJvZHkiLGJveCksTz1bIkFjY3VyYWN5IiwiUHJlY2lzaW9uIiwiUmVjYWxsIiwiRjEiXTsKIGZ1bmN0aW9uIHNob3coKXt2YXIgcz1TW2ldO2JvZHkuaW5uZXJIVE1MPSc8cCBjbGFzcz0iY2Qtc21hbGwiPlNjZW5hcmlvICcrKGkrMSkrJyBvZiAnK1MubGVuZ3RoKycgwrcgU2NvcmUgJytzY29yZSsnPC9wPjxkaXYgY2xhc3M9ImNkLWdhbWUtY2FyZCIgc3R5bGU9Im1pbi1oZWlnaHQ6NjRweCI+JytzLnQrJzwvZGl2PjxwIGNsYXNzPSJjZC1zbWFsbCIgc3R5bGU9Im1hcmdpbjo4cHggMCAwIj5XaGljaCBtZXRyaWMgc2hvdWxkIHlvdSBmb2N1cyBvbj88L3A+PGRpdiBjbGFzcz0iY2Qtcm93IiBzdHlsZT0ibWFyZ2luLXRvcDo4cHgiPicrTy5tYXAoZnVuY3Rpb24obyl7cmV0dXJuICc8YnV0dG9uIHR5cGU9ImJ1dHRvbiIgY2xhc3M9ImNkLWJ0biBjZC13aC1vcHQiPicrbysnPC9idXR0b24+J30pLmpvaW4oIiIpKyc8L2Rpdj48ZGl2IGNsYXNzPSJjZC1mZWVkYmFjayIgaGlkZGVuPjwvZGl2PjxkaXYgc3R5bGU9Im1hcmdpbi10b3A6MTBweCI+PGJ1dHRvbiB0eXBlPSJidXR0b24iIGNsYXNzPSJjZC1idG4gY2QtcHJpbWFyeSBjZC13aC1uZXh0IiBoaWRkZW4+JysoaTxTLmxlbmd0aC0xPyJOZXh0IHNjZW5hcmlvIjoiU2VlIG15IHNjb3JlIikrJzwvYnV0dG9uPjwvZGl2Pic7CiAgJCQoIi5jZC13aC1vcHQiLGJvZHkpLmZvckVhY2goZnVuY3Rpb24oYil7Yi5hZGRFdmVudExpc3RlbmVyKCJjbGljayIsZnVuY3Rpb24oKXt2YXIgb2s9Yi50ZXh0Q29udGVudD09PXMuYTtpZihvaylzY29yZSsrOyQkKCIuY2Qtd2gtb3B0Iixib2R5KS5mb3JFYWNoKGZ1bmN0aW9uKHgpe3guZGlzYWJsZWQ9dHJ1ZTtpZih4LnRleHRDb250ZW50PT09cy5hKXguY2xhc3NMaXN0LmFkZCgiY2QtcmlnaHQiKTtlbHNlIGlmKHg9PT1iKXguY2xhc3NMaXN0LmFkZCgiY2Qtd3JvbmciKX0pO3ZhciBmYj0kKCIuY2QtZmVlZGJhY2siLGJvZHkpO2ZiLmhpZGRlbj1mYWxzZTtmYi5jbGFzc05hbWU9ImNkLWZlZWRiYWNrICIrKG9rPyJjZC1vayI6ImNkLW5vIik7ZmIuaW5uZXJIVE1MPSI8Yj4iKyhvaz8iQ29ycmVjdC4iOiJCZXN0IGFuc3dlcjogIitzLmErIi4iKSsiPC9iPiAiK3MuZTskKCIuY2Qtd2gtbmV4dCIsYm9keSkuaGlkZGVuPWZhbHNlfSl9KTsKICAkKCIuY2Qtd2gtbmV4dCIsYm9keSkuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7aSsrO2lmKGk8Uy5sZW5ndGgpc2hvdygpO2Vsc2V7Ym9keS5pbm5lckhUTUw9JzxkaXYgY2xhc3M9ImNkLWZlZWRiYWNrIGNkLW9rIj48Yj5Zb3Ugc2NvcmVkICcrc2NvcmUrJyBvdXQgb2YgJytTLmxlbmd0aCsnLjwvYj4gVGhlIHJ1bGUgb2YgdGh1bWI6IGFzayB3aGljaCBtaXN0YWtlIGNvc3RzIG1vcmUsIGEgZmFsc2UgcG9zaXRpdmUgb3IgYSBmYWxzZSBuZWdhdGl2ZSwgYW5kIHBpY2sgdGhlIG1ldHJpYyB0aGF0IHB1bmlzaGVzIGl0LjwvZGl2PjxkaXYgc3R5bGU9Im1hcmdpbi10b3A6MTBweCI+PGJ1dHRvbiB0eXBlPSJidXR0b24iIGNsYXNzPSJjZC1idG4gY2Qtd2gtYWdhaW4iPlBsYXkgYWdhaW48L2J1dHRvbj48L2Rpdj4nOyQoIi5jZC13aC1hZ2FpbiIsYm9keSkuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7aT0wO3Njb3JlPTA7c2hvdygpfSl9fSl9CiAkKCIuY2Qtd2gtc3RhcnQiLGJveCkuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLHNob3cpOwp9KTsKCi8qIDguIFF1aXogKi8KJCQoIi5jZGFpLmNkLWNtLXF1aXoiKS5mb3JFYWNoKGZ1bmN0aW9uKGJveCl7CiB2YXIgUT1bCiB7cToiQSBtb2RlbCBmbGFncyA1MCB0cmFuc2FjdGlvbnMgYXMgZnJhdWQuIDQwIHJlYWxseSBhcmUgZnJhdWQuIFdoYXQgaXMgaXRzIHByZWNpc2lvbj8iLG86WyI0MCUiLCI4MCUiLCI1MCUiLCJDYW5ub3QgdGVsbCJdLGE6MSxlOiJQcmVjaXNpb24gPSBUUCAvIChUUCArIEZQKSA9IDQwIC8gNTAgPSA4MCUuIFlvdSBkbyBub3QgbmVlZCB0byBrbm93IGhvdyBtYW55IGZyYXVkcyB3ZXJlIG1pc3NlZCB0byBjb21wdXRlIHByZWNpc2lvbi4ifSwKIHtxOiJUaGVyZSB3ZXJlIDEwMCByZWFsIGZyYXVkIGNhc2VzIGFuZCB0aGUgbW9kZWwgY2F1Z2h0IDQwLiBXaGF0IGlzIGl0cyByZWNhbGw\/IixvOlsiNDAlIiwiODAlIiwiNjAlIiwiMTAwJSJdLGE6MCxlOiJSZWNhbGwgPSBUUCAvIChUUCArIEZOKSA9IDQwIC8gMTAwID0gNDAlLiJ9LAoge3E6IlVzaW5nIHRob3NlIHR3byBhbnN3ZXJzIChwcmVjaXNpb24gMC44LCByZWNhbGwgMC40KSwgd2hhdCBpcyB0aGUgRjEgc2NvcmU\/IixvOlsiMC42MCIsIjAuNTMiLCIwLjQwIiwiMC44MCJdLGE6MSxlOiJGMSA9IDIgw5cgMC44IMOXIDAuNCAvICgwLjggKyAwLjQpID0gMC42NCAvIDEuMiDiiYggMC41MzMuIFRoZSBoYXJtb25pYyBtZWFuIHNpdHMgY2xvc2VyIHRvIHRoZSB3ZWFrZXIgbWV0cmljLiJ9LAoge3E6IkEgVHlwZSBJSSBlcnJvciBpcyB0aGUgc2FtZSBhcyBhLi4uIixvOlsiRmFsc2UgcG9zaXRpdmUiLCJGYWxzZSBuZWdhdGl2ZSIsIlRydWUgbmVnYXRpdmUiLCJXcm9uZyB0aHJlc2hvbGQiXSxhOjEsZToiVHlwZSBJID0gZmFsc2UgcG9zaXRpdmUgKGEgZmFsc2UgYWxhcm0pLiBUeXBlIElJID0gZmFsc2UgbmVnYXRpdmUgKGEgbWlzcykuIn0sCiB7cToiWW91IGxvd2VyIHRoZSBjbGFzc2lmaWNhdGlvbiB0aHJlc2hvbGQgZnJvbSAwLjUgdG8gMC4yLiBXaGF0IHVzdWFsbHkgaGFwcGVucz8iLG86WyJQcmVjaXNpb24gdXAsIHJlY2FsbCBkb3duIiwiUmVjYWxsIHVwLCBwcmVjaXNpb24gZG93biIsIkJvdGggZ28gdXAiLCJOb3RoaW5nIGNoYW5nZXMiXSxhOjEsZToiTW9yZSBleGFtcGxlcyBhcmUgcHJlZGljdGVkIHBvc2l0aXZlLCBzbyB5b3UgY2F0Y2ggbW9yZSByZWFsIHBvc2l0aXZlcyAocmVjYWxsIHVwKSBhbmQgYWxzbyBmbGFnIG1vcmUgbmVnYXRpdmVzIChwcmVjaXNpb24gZG93bikuIn0sCiB7cToiQSBtb2RlbCBoYXMgUk9DIEFVQyA9IDAuNS4gV2hhdCBkb2VzIHRoYXQgbWVhbj8iLG86WyJJdCBpcyA1MCUgYWNjdXJhdGUiLCJJdCByYW5rcyBubyBiZXR0ZXIgdGhhbiByYW5kb20gZ3Vlc3NpbmciLCJJdCBpcyBwZXJmZWN0IGF0IHRocmVzaG9sZCAwLjUiLCJIYWxmIGl0cyBwcmVkaWN0aW9ucyBhcmUgcG9zaXRpdmUiXSxhOjEsZToiQVVDIGlzIHRoZSBwcm9iYWJpbGl0eSB0aGF0IGEgcmFuZG9tIHBvc2l0aXZlIGdldHMgYSBoaWdoZXIgc2NvcmUgdGhhbiBhIHJhbmRvbSBuZWdhdGl2ZS4gMC41IG1lYW5zIHRoZSBzY29yZXMgY2Fycnkgbm8gdXNlZnVsIGluZm9ybWF0aW9uLiJ9XTsKIHZhciBpPTAsc2NvcmU9MCxib2R5PSQoIi5jZC1jcS1ib2R5Iixib3gpOwogZnVuY3Rpb24gc2hvdygpe3ZhciBpdD1RW2ldO2JvZHkuaW5uZXJIVE1MPSc8cCBjbGFzcz0iY2Qtc21hbGwiPlF1ZXN0aW9uICcrKGkrMSkrJyBvZiAnK1EubGVuZ3RoKycgwrcgU2NvcmUgJytzY29yZSsnPC9wPjxkaXYgY2xhc3M9ImNkLWdhbWUtY2FyZCIgc3R5bGU9Im1pbi1oZWlnaHQ6NjRweCI+JytpdC5xKyc8L2Rpdj48ZGl2IGNsYXNzPSJjZC1xLW9wdHMiPicraXQuby5tYXAoZnVuY3Rpb24obyxrKXtyZXR1cm4gJzxidXR0b24gdHlwZT0iYnV0dG9uIiBjbGFzcz0iY2QtYnRuIiBkYXRhLWs9IicraysnIj4nK28rJzwvYnV0dG9uPid9KS5qb2luKCIiKSsnPC9kaXY+PGRpdiBjbGFzcz0iY2QtZmVlZGJhY2siIGhpZGRlbj48L2Rpdj48ZGl2IHN0eWxlPSJtYXJnaW4tdG9wOjEycHgiPjxidXR0b24gdHlwZT0iYnV0dG9uIiBjbGFzcz0iY2QtYnRuIGNkLXByaW1hcnkgY2QtY3EtbmV4dCIgaGlkZGVuPicrKGk8US5sZW5ndGgtMT8iTmV4dCBxdWVzdGlvbiI6IlNlZSBteSBzY29yZSIpKyc8L2J1dHRvbj48L2Rpdj4nOwogICQkKCIuY2QtcS1vcHRzIGJ1dHRvbiIsYm9keSkuZm9yRWFjaChmdW5jdGlvbihidCl7YnQuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7dmFyIGs9K2J0LmRhdGFzZXQuayxvaz1rPT09aXQuYTtpZihvaylzY29yZSsrOyQkKCIuY2QtcS1vcHRzIGJ1dHRvbiIsYm9keSkuZm9yRWFjaChmdW5jdGlvbih4LGope3guZGlzYWJsZWQ9dHJ1ZTtpZihqPT09aXQuYSl4LmNsYXNzTGlzdC5hZGQoImNkLXJpZ2h0Iik7ZWxzZSBpZihqPT09ayl4LmNsYXNzTGlzdC5hZGQoImNkLXdyb25nIil9KTt2YXIgZmI9JCgiLmNkLWZlZWRiYWNrIixib2R5KTtmYi5oaWRkZW49ZmFsc2U7ZmIuY2xhc3NOYW1lPSJjZC1mZWVkYmFjayAiKyhvaz8iY2Qtb2siOiJjZC1ubyIpO2ZiLmlubmVySFRNTD0iPGI+Iisob2s\/IkNvcnJlY3QuIjoiTm90IHF1aXRlLiIpKyI8L2I+ICIraXQuZTskKCIuY2QtY3EtbmV4dCIsYm9keSkuaGlkZGVuPWZhbHNlfSl9KTsKICAkKCIuY2QtY3EtbmV4dCIsYm9keSkuYWRkRXZlbnRMaXN0ZW5lcigiY2xpY2siLGZ1bmN0aW9uKCl7aSsrO2lmKGk8US5sZW5ndGgpc2hvdygpO2Vsc2V7Ym9keS5pbm5lckhUTUw9JzxkaXYgY2xhc3M9ImNkLWZlZWRiYWNrIGNkLW9rIj48Yj5Zb3Ugc2NvcmVkICcrc2NvcmUrJyBvdXQgb2YgJytRLmxlbmd0aCsnLjwvYj4gJysoc2NvcmU+PTU\/IllvdSBjYW4gcmVhZCBhbmQgZXhwbGFpbiBhbnkgY29uZnVzaW9uIG1hdHJpeC4iOiJSZXZpc2l0IHRoZSBtZXRyaWNzIGNhbGN1bGF0b3IgYW5kIHRocmVzaG9sZCBleHBsb3JlciwgdGhlbiB0cnkgYWdhaW4uIikrJzwvZGl2PjxkaXYgc3R5bGU9Im1hcmdpbi10b3A6MTBweCI+PGJ1dHRvbiB0eXBlPSJidXR0b24iIGNsYXNzPSJjZC1idG4gY2QtY3EtYWdhaW4iPlRyeSBhZ2FpbjwvYnV0dG9uPjwvZGl2Pic7JCgiLmNkLWNxLWFnYWluIixib2R5KS5hZGRFdmVudExpc3RlbmVyKCJjbGljayIsZnVuY3Rpb24oKXtpPTA7c2NvcmU9MDtzaG93KCl9KX19KX0KICQoIi5jZC1jcS1zdGFydCIsYm94KS5hZGRFdmVudExpc3RlbmVyKCJjbGljayIsc2hvdyk7Cn0pOwoKLyogY29weSBidXR0b25zICovCiQkKCIuY2RhaS5jZC1jb2RlYm94IikuZm9yRWFjaChmdW5jdGlvbihib3gpe3ZhciBidD0kKCIuY2QtY29weSIsYm94KTtpZighYnQpcmV0dXJuO2J0LmFkZEV2ZW50TGlzdGVuZXIoImNsaWNrIixmdW5jdGlvbigpe3ZhciB0PSQoInByZSBjb2RlIixib3gpLnRleHRDb250ZW50O2Z1bmN0aW9uIG9rKCl7YnQudGV4dENvbnRlbnQ9IkNvcGllZCI7c2V0VGltZW91dChmdW5jdGlvbigpe2J0LnRleHRDb250ZW50PSJDb3B5In0sMTUwMCl9ZnVuY3Rpb24gc2VsKCl7dmFyIHI9ZG9jdW1lbnQuY3JlYXRlUmFuZ2UoKTtyLnNlbGVjdE5vZGVDb250ZW50cygkKCJwcmUgY29kZSIsYm94KSk7dmFyIHM9d2luZG93LmdldFNlbGVjdGlvbigpO3MucmVtb3ZlQWxsUmFuZ2VzKCk7cy5hZGRSYW5nZShyKTtidC50ZXh0Q29udGVudD0iUHJlc3MgQ3RybCtDIn1pZihuYXZpZ2F0b3IuY2xpcGJvYXJkJiZuYXZpZ2F0b3IuY2xpcGJvYXJkLndyaXRlVGV4dCl7bmF2aWdhdG9yLmNsaXBib2FyZC53cml0ZVRleHQodCkudGhlbihvayxzZWwpfWVsc2Ugc2VsKCl9KX0pOwp9KSgpOwo=\"><\/script>\n","protected":false},"excerpt":{"rendered":"<p>Learn the confusion matrix by doing: sort predictions into TP, FP, FN and TN, calculate precision, recall and F1 step by step, explore ROC and AUC with a live threshold, and read a 3&#215;3 matrix, with Python code.<\/p>\n","protected":false},"author":1,"featured_media":1178,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[47],"tags":[],"class_list":["post-1179","post","type-post","status-publish","format-standard","has-post-thumbnail","category-ai-machine-learning"],"featured_image_src":"https:\/\/codeanddebug.in\/blog\/wp-content\/uploads\/2026\/10\/confusion-matrix-in-machine-learning.jpg","author_info":{"display_name":"codeanddebug","author_link":"https:\/\/codeanddebug.in\/blog\/author\/codeanddebug\/"},"_links":{"self":[{"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/posts\/1179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/comments?post=1179"}],"version-history":[{"count":2,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/posts\/1179\/revisions"}],"predecessor-version":[{"id":1183,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/posts\/1179\/revisions\/1183"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/media\/1178"}],"wp:attachment":[{"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/media?parent=1179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/categories?post=1179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codeanddebug.in\/blog\/wp-json\/wp\/v2\/tags?post=1179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}