Background

Problem Statement: serving a unionml app today consists of a hodge-podge of APIs and different conventions for deploying services that’s cobbled together using different technologies (FastAPI, Mangum, SAM CLI), each with different conventions around app creation, builds, and deployment.

It’s on the user to figure out how to put all of these technologies together to achieve what they want.

Objective: Make a consistent, logical experience for deploying and serving a unionml app

Current state:

• Deploying a Flyte backend for training and batch prediction, requires doing model.remote(...) , then model.remote_deploy()
• Serving a FastAPI app requires app = FastAPI(); model.serve(app)
• Serving a AWS Lambda serverless app requires app = FastAPI(); model.serve(app); handler = Mangum(App)
• Reacting to S3 events to trigger AWS Lambda function requires knowledge of sam cli and creating a custom handler.

What’s missing:

• There’s no unionml-first way of scheduling a job on a Flyte backend, users have to create the Flyte LaunchPlans themselves.

Proposal

1. Create an extensible ModelService class that defines a protocol for the following types of services for model prediction (and potentially others like model explanations, training, tuning, active learning etc.):
   1. API/Function-based prediction microservices
   2. Scheduling-based microservices
   3. Event-based reactive microservices
2. Create Unionml-native (i.e. Flyte-native) scheduling for prediction, training, and evaluation (back-testing).
3. Abstract away the build and deployment process for the ModelService microservices.
   1. For the ModelService class, a framework-specific build and deploy process hides the complexity so that users don’t have to, e.g. learn the AWS SAM cli

Native Scheduling