Identify relations between entities, calibrate confidence and abstain on unknown relations. Extended from ENSAI work with Maxime Ferret.
Key metric
89.16%
Macro-F1 · known-relation test in the 56-known open-set protocol
Illustrative example · relation extraction
Marie CurieHEAD discovered poloniumTAIL.
open-setKNOWNUNKNOWN → abstain
A calibrated decision, with the option to abstain.
Architecture
RelationScope NLP
Components, data flows and trust boundaries of the system.
Scroll the diagram →
Technologies
PyTorch
ModernBERT · HF
FastAPI
Next.js
TypeScript
Cloud Run
Cloud Storage
Cloud Tasks
Terraform
Docker
GitHub Actions
Design decisions
01
Only the Next.js app is public; it obtains a Google ID token server-side for the private API, so the browser never sees a credential.
02
Two buckets with distinct permissions: read-only for models, read/write for telemetry and jobs, with 90-day retention.
03
Explicit lifecycle: train → frozen evaluation → promotion gates → register → promote. Production traffic never triggers retraining.
04
Privacy by default: no sentence or entity text in logs; raw feedback text only with explicit consent.
What I built
Entity markers, pair-aware pooling, temperature calibration and abstention thresholds. FastAPI/Next.js serving, word-occlusion explanations, human review and durable asynchronous processing.
Result and trade-off
Balanced profile: 84.45% unknown rejection and 61.25% known acceptance. The protocol reserves 16 unseen relations for the final test, separate from calibration.
Scope and limitations
Occlusion provides sensitivity evidence, not a causal explanation. Classifier metrics do not measure automatic entity-detection quality. V3 additions are presented separately from the V1 benchmark.
