Inference Endpoints

Inference Endpoints offers a secure production solution to easily deploy any Transformers, Sentence-Transformers and Diffusion models from the Hub on dedicated and autoscaling infrastructure managed by Model Database.

A Model Database Endpoint is built from a Model Database Model Repository. When an Endpoint is created, the service creates image artifacts that are either built from the model you select or a custom-provided container image. The image artifacts are completely decoupled from the Model Database Hub source repositories to ensure the highest security and reliability levels.

Inference Endpoints support all of the Transformers, Sentence-Transformers and Diffusion tasks as well as custom tasks not supported by Transformers yet like speaker diarization and diffusion.

In addition, Inference Endpoints gives you the option to use a custom container image managed on an external service, for instance, Docker Hub, AWS ECR, Azure ACR, or Google GCR.

Documentation and Examples

• Security & Compliance
• Supported Transformers Task
• API Reference
• Autoscaling
• FAQ
• Help & Support

Guides

• Access the solution (UI)
• Create your first Endpoint
• Send Requests to Endpoints
• Update your Endpoint
• Advanced Setup (Instance Types, Auto Scaling, Versioning)
• Create a Private Endpoint with AWS PrivateLink
• Add custom Dependencies
• Create custom Inference Handler
• Use a custom Container Image
• Access and read Logs
• Access and view Metrics
• Change Organization or Account

Others

• Inference Endpoints Versions
• Serialization & Deserialization for Requests