Provider Development Guide
3. Provider Development Guide¶
This guide is designed to walk you through the process of extending the SLOP Server by adding your own custom providers (pkg/providers/<unique_provider_name>
). It details the key steps, from initial setup to testing your new provider.
Setting Up¶
Before you begin developing a new provider, ensure your Go development environment is correctly configured and you have access to the SLOP Server project source code. It is recommended to familiarize yourself with the overall provider architecture described in the Core Concepts section of this document.
The initial steps for setting up your provider are as follows:
- Choose a Provider Name: Select a unique and descriptive name for your provider, adhering to Go naming conventions (lowercase, single word, no hyphens or spaces). For example,
myprovider
. - Create Directory Structure: In the
pkg/providers/
directory of your SLOP Server project, create a new directory named after your provider (e.g.,pkg/providers/myprovider/
). - Go Module Initialization (if necessary): If your provider has specific dependencies not managed by the main project module, you might need to manage its dependencies. However, in most cases, dependencies will be handled at the main project level.
- Prepare Internal Dependencies: Your provider will likely interact with several core components of the SLOP Server. Ensure you understand how to import and use the interfaces and structs provided by internal packages such as
internal/providers
,internal/core
,internal/jobmanager
,internal/pubsub
, andinternal/resources
.
The slop-server-cli provider create <provider_name>
CLI tool can be used to automatically generate a basic structure for your new provider, including initial files and the models
and services
directories. This will provide a solid starting point for your development.
Basic Provider Structure¶
A typical provider within the SLOP Server is organized modularly to encapsulate its logic and dependencies. While the structure might vary slightly based on the provider's complexity, here are the common components found in the pkg/providers/<provider_name>/
directory:
-
adapter.go
: This is the heart of the provider. This file contains the implementation of the main adapter struct (e.g.,MyProviderAdapter
) that satisfies theproviders.Provider
interface (defined ininternal/providers/interfaces.go
). It handles initialization, configuration, registration of capabilities (tools, webhooks, etc.), and interaction with the SLOP Server core. It usually includes theinit()
function to register the provider's factory with the central registry. -
client.go
(Optional): If your provider interacts with an external API or service, this file will contain the HTTP client logic, methods for making requests, handling authentication with the third-party service, and parsing responses. Theunipile
example shows aUnipileClient
with its own services. -
models/
(Directory, Optional): This directory contains the Go struct definitions specific to your provider's domain. These models can represent data exchanged with an external service, specific configurations, or entities manipulated by the provider. For example, inunipile
,account.go
,email.go
, etc., are found inpkg/providers/unipile/models/
. -
services/
(Directory, Optional): For more complex providers, this directory can organize business logic into different services. Each service can then expose a set of capabilities (which will become tools). Theunipile
example uses this pattern withaccount_service.go
,email_service.go
, etc., inpkg/providers/unipile/services/
. Each service file typically defines a service struct and aGetCapabilities()
method. -
*_webhook_handler.go
(Optional): If your provider needs to receive webhooks from external services, the HTTP handlers for these webhooks are implemented here. These handlers are then registered via the adapter'sGetRoutes()
method. For example,email_webhook_handler.go
inunipile
. -
*_job.go
(Optional): For asynchronous tasks or background processing, this type of file defines the job logic. These jobs are then registered and managed by the centralJobManager
.sync_job.go
inunipile
is an example. -
store.go
orpg_store.go
(Optional): If your provider requires specific data persistence (beyond what theCoreStore
orResourceManager
offer globally), you can define a store interface (e.g.,MyProviderStoreInterface
) and its implementation (often for PostgreSQL, hencepg_store.go
). Theunipile
example has apg_store.go
for its specific storage needs. -
auth.go
(Optional): May contain logic specific to the provider's own authentication or managing authentication mechanisms for the external services it uses.unipile
has anauth.go
for handling authentication links. -
helper.go
(Optional): Utility functions specific to the provider that don't fit into other categories. -
readme.md
: A Markdown file describing the provider, its purpose, configuration, capabilities, and any relevant development notes. This file is crucial for the provider's maintainability and understanding.
The unipile_pkg_files.txt
example clearly illustrates this structure with files like adapter.go
, client.go
, and the models/
and services/
directories.
Implementing the Adapter¶
The adapter is the central component of your provider. It's a Go struct that implements the providers.Provider
interface defined in internal/providers/interfaces.go
. This interface defines the contract all providers must adhere to for proper integration with the SLOP Server.
Key methods of the providers.Provider
interface your adapter must implement:
GetID() string
: Returns the unique identifier for your provider (e.g.,"myprovider"
). This ID is used for registration and configuration.GetName() string
: Returns a human-readable name for your provider (e.g.,"My Provider Integration"
).GetDescription() string
: Returns a detailed description of the provider's role and functionalities.GetCompactDescription() string
: Returns a short, concise description of the provider.Initialize(config map[string]interface{}) error
: This method is called by the SLOP Server at startup to configure the provider. It receives a configuration map (config
) extracted from the server's global configuration file (e.g.,config.yaml
) or environment variables. Here, you should:- Validate dependencies injected during adapter creation (via the factory).
- Read and validate provider-specific configuration parameters (API keys, base URLs, etc.).
- Initialize external service clients, provider-specific database connections, etc.
- Mark the provider as initialized (usually via an internal boolean
initialized
). Theunipile/adapter.go
example shows how to loadbase_url
,api_key
, andclient_secret
from configuration or environment variables.
GetProviderTools() []providers.ProviderTool
: Returns a list of all tools (capabilities) exposed by this provider. Each tool is defined by aproviders.ProviderTool
struct. This method is typically called afterInitialize
.ExecuteTool(ctx context.Context, toolID string, params map[string]interface{}) (interface{}, error)
: Executes a specific tool identified bytoolID
with the providedparams
. ThetoolID
usually follows a format likeproviderID_serviceName_capabilityName
orproviderID_toolName
.GetRoutes() []providers.Route
(Optional): If your provider exposes HTTP endpoints (e.g., for webhooks), this method returns a list ofproviders.Route
.Shutdown() error
(Optional): Allows the provider to cleanly release its resources (close connections, stop goroutines, etc.) when the server shuts down.
In addition to providers.Provider
, your adapter can also implement other optional interfaces from the internal/providers
package to extend its functionality:
SchemaInitializer
: If your provider needs to create or migrate a specific database schema during initialization. You would then implementInitializeSchema(ctx context.Context, pool *pgxpool.Pool) error
.EventInfoProvider
: If your provider publishes specific events, it can implementGetEventDefinitions() []providers.EventDefinition
to declare them.
Factory and init()
Registration
Each provider must also supply a "factory" function (e.g., newMyProviderAdapterFactory
) that takes providers.ProviderDependencies
as an argument and returns an instance of providers.Provider
(your adapter) and an error. These dependencies (ProviderDependencies
) include core services like CoreStore
, PgxPool
, EncryptionKey
, ResourceManager
, Publisher
(for Pub/Sub), and JobManager
.
This factory is then registered in the init()
function of your adapter.go
file by calling providers.RegisterFactory(ProviderID, newMyProviderAdapterFactory)
. This allows the SLOP Server to discover and instantiate your provider at startup.
The unipile/adapter.go
example shows a UnipileAdapter
struct with its dependencies, its newUnipileAdapterFactory
factory, and the implementation of the providers.Provider
interface methods.
Registering the Provider¶
For your provider to be recognized and usable by the SLOP Server, it must be explicitly registered during the server's initialization phase. This process involves two main steps:
-
Creating a Provider Factory: In your
adapter.go
file, you must define a "factory" function. This function is responsible for creating an instance of your provider adapter. It takes aproviders.ProviderDependencies
struct as an argument, which contains all the core dependencies your provider might need (likeCoreStore
,PgxPool
,EncryptionKey
,ResourceManager
,Publisher
,JobManager
). The factory must return an instance ofproviders.Provider
(i.e., your adapter) and anerror
.Example factory signature (from
unipile/adapter.go
):// newUnipileAdapterFactory is the factory function called by the registry. func newUnipileAdapterFactory(deps providers.ProviderDependencies) (providers.Provider, error) { // ... dependency checks ... adapter, err := NewUnipileAdapter(deps.CoreStore, deps.EncryptionKey, deps.PgxPool, deps.ResourceManager, deps.Publisher, deps.JobManager) // ... error handling ... return adapter, nil }
-
Registration via
init()
function: Still in youradapter.go
, you will use Go's specialinit()
function to register your factory with the SLOP Server's central provider registry. This is done by callingproviders.RegisterFactory(ProviderID, yourFactoryFunction)
. TheProviderID
must be a unique string constant identifying your provider (e.g.,const ProviderID = "myprovider"
).Example registration (from
unipile/adapter.go
):const ( ProviderID = "unipile" ) func init() { log.Println("DEBUG: unipile init() registering factory") if err := providers.RegisterFactory(ProviderID, newUnipileAdapterFactory); err != nil { log.Fatalf("CRITICAL: Failed to register provider factory for '%s': %v", ProviderID, err) } log.Printf("DEBUG: Provider factory for '%s' registration submitted.", ProviderID) }
Once these steps are completed, the SLOP Server will be able to instantiate and initialize your provider at startup. The server's initialization process (GetAllProviders()
, then calling Initialize()
on each provider) will ensure your provider is ready to operate.
Defining and Registering Tools¶
Tools are discrete functionalities exposed by your provider. They can be called by users via the API, by LLMs, or by other system components. Defining and registering tools primarily happens via your adapter's GetProviderTools()
method.
Tool Structure (providers.ProviderTool
)
Each tool is represented by the providers.ProviderTool
struct, which includes the following fields:
ID
: A unique identifier for the tool, typically in the formatproviderID_serviceName_capabilityName
orproviderID_toolName
(e.g.,"myprovider_email_send"
).Name
: A human-readable name for the tool (e.g.,"Send Email"
).Scope
: Usually an empty string""
for local providers.Description
: A clear and concise explanation of what the tool does, its effects, and when to use it. This description is crucial for LLMs.Parameters
: Amap[string]providers.Parameter
describing the parameters the tool accepts. Eachproviders.Parameter
defines the type (string
,integer
,boolean
,object
,array
), description, requirement status (Required
), category (body
,query
,path
), etc.Examples
: A list ofproviders.ToolExample
illustrating how to use the tool with sample parameters and expected results.
Implementing GetProviderTools()
In your adapter's GetProviderTools()
method, you will construct and return a slice ([]providers.ProviderTool
) of all tools offered by your provider.
If you use a service-based architecture (as in unipile
with its services/
directory), each service might have a GetCapabilities() []providers.ProviderCapability
method. The adapter's GetProviderTools()
method would then iterate over these services, retrieve their capabilities, and transform them into providers.ProviderTool
.
Example (conceptual, based on unipile/adapter.go
):
func (a *MyProviderAdapter) GetProviderTools() []providers.ProviderTool {
if !a.initialized {
slog.Warn("Attempted to get tools from uninitialized provider", "provider", a.GetID())
return nil
}
var tools []providers.ProviderTool
providerID := a.GetID()
// If you have an internal service registry, like in UnipileClient
// internalServices := a.client.Registry.GetAll()
// for serviceName, service := range internalServices {
// for _, capability := range service.GetCapabilities() { // GetCapabilities() is a method of your service
// tool := providers.ProviderTool{
// ID: fmt.Sprintf("%s_%s_%s", providerID, serviceName, capability.Name),
// Name: capability.Name,
// Description: capability.Description,
// Parameters: capability.Parameters, // map[string]providers.Parameter
// Examples: capability.Examples,
// }
// tools = append(tools, tool)
// }
// }
// Or define tools directly
tools = append(tools, providers.ProviderTool{
ID: fmt.Sprintf("%s_my_feature", providerID),
Name: "My Specific Feature",
Description: "Performs a specific action for my provider.",
Parameters: map[string]providers.Parameter{
"param1": {
Type: "string",
Description: "Description of parameter 1.",
Required: true,
Category: "body",
},
},
})
slog.Debug("Returning tools for provider", "provider", providerID, "count", len(tools))
return tools
}
Tool Parameters (providers.Parameter
)
Precise parameter definition is essential. The providers.Parameter
struct includes:
Type
: Data type (string
,integer
,boolean
,number
,array
,object
).Description
: Explanation of the parameter.Required
: Boolean indicating if the parameter is mandatory.Category
: Where the parameter is expected in an HTTP request (body
,query
,path
,header
). For tools called by LLMs or internally,body
is often used by convention for a set of parameters.Enum
: List of possible values if the parameter is an enumeration.Default
: Default value if the parameter is not provided.Items
: IfType
isarray
,Items
(of type*providers.Parameter
) describes the type of array elements.Properties
: IfType
isobject
,Properties
(of typemap[string]providers.Parameter
) describes the object's fields.
A clear definition of tools and their parameters enables seamless integration with the rest of the SLOP Server and facilitates their use by LLMs.
Handling Tool Execution¶
Once tools are defined and registered, their execution logic must be implemented in your adapter's ExecuteTool(ctx context.Context, toolID string, params map[string]interface{}) (interface{}, error)
method.
This method is responsible for:
- Checking Initialization: Ensure the provider is initialized before attempting to execute a tool.
- Identifying the Tool: Parse
toolID
to determine which specific tool needs to be executed. ThetoolID
usually follows a format likeproviderID_serviceName_capabilityName
orproviderID_toolName
. You'll need to extract the service name (if applicable) and the capability/tool name. - Validating and Extracting Parameters: The
params
are provided as amap[string]interface{}
. You must extract the expected values, convert them to the correct types, and validate their presence if mandatory, as well as their format. - Executing Business Logic: Call the internal function or method that actually implements the tool's functionality. Pass the
context.Context
and validated parameters to this function. - Returning the Result: The method must return two values:
interface{}
for the execution result (which will typically be serialized to JSON) and anerror
if the execution failed.
Example structure for ExecuteTool
(based on unipile/adapter.go
):
func (a *MyProviderAdapter) ExecuteTool(ctx context.Context, toolID string, params map[string]interface{}) (interface{}, error) {
if !a.initialized {
return nil, fmt.Errorf("provider %s not initialized", a.GetID())
}
slog.InfoContext(ctx, "Executing tool", "provider", a.GetID(), "toolID", toolID)
// Parse toolID. Example for "providerID_serviceName_capabilityName"
parts := strings.SplitN(toolID, "_", 3)
if len(parts) < 2 || parts[0] != a.GetID() { // At least providerID_toolName
return nil, fmt.Errorf("invalid or mismatched tool ID format for provider %s: %s", a.GetID(), toolID)
}
var serviceName, capabilityName string
if len(parts) == 3 {
serviceName = parts[1]
capabilityName = parts[2]
} else { // len(parts) == 2, format providerID_toolName
capabilityName = parts[1]
// serviceName remains empty, or you have a convention for direct tools
}
// Route to specific tool logic
// If using an internal services architecture:
// internalService := a.client.Registry.GetService(serviceName) // Hypothetical
// if internalService == nil {
// return nil, fmt.Errorf("unknown service '%s' for provider %s", serviceName, a.GetID())
// }
// return internalService.ExecuteCapability(ctx, capabilityName, params)
// Or a switch on capabilityName (or a combination of serviceName/capabilityName)
switch capabilityName { // or toolID if the format is simpler
case "my_feature":
// Extract and validate parameters from 'params'
param1, ok := params["param1"].(string)
if !ok || param1 == "" {
return nil, errors.New("missing or invalid 'param1' parameter")
}
return a.doMyFeature(ctx, param1) // Call the implementation function
// case "another_tool":
// return a.doAnotherTool(ctx, params)
default:
return nil, fmt.Errorf("unknown tool ID or capability '%s' (service: '%s') for provider %s", capabilityName, serviceName, a.GetID())
}
}
// Implementation function for the "my_feature" tool
func (a *MyProviderAdapter) doMyFeature(ctx context.Context, param1 string) (interface{}, error) {
slog.InfoContext(ctx, "Executing my_feature", "param1", param1)
// ... tool's business logic ...
result := map[string]interface{}{
"status": "success",
"message": "Feature executed with " + param1,
}
return result, nil
}
Proper error handling and returning clear, informative error messages are crucial. Parameter validation can be facilitated by libraries like github.com/mitchellh/mapstructure
to decode the parameter map into a tool-specific Go struct, allowing the use of validation tags.
Implementing Webhook Handlers¶
If your provider needs to receive asynchronous notifications from external services (e.g., a status update, a new event), you will need to implement webhook handlers. These handlers are HTTP endpoints that the external service will call.
Key Steps:
-
Define Routes: In your adapter's
GetRoutes() []providers.Route
method, you declare the routes for your webhooks. Eachproviders.Route
specifies:Path
: The URL path for the webhook (e.g.,"/webhooks/myprovider/notification"
).Method
: The expected HTTP method (usually"POST"
).Handler
: An instance ofhttp.Handler
(oftenhttp.HandlerFunc(yourHandlerFunction)
).Description
: A brief description of what the webhook does.Auth
: A boolean indicating if this route requires SLOP Server authentication. For external webhooks, this is typicallyfalse
.
Example
GetRoutes
(inspired byunipile/adapter.go
):func (a *MyProviderAdapter) GetRoutes() []providers.Route { return []providers.Route{ { Path: fmt.Sprintf("/webhooks/%s/notify", a.GetID()), Method: "POST", Handler: http.HandlerFunc(a.handleWebhookNotification), // Your handler function Description: "Handles incoming notifications from MyProvider.", Auth: false, }, } }
-
Implement the Handler Function: Create a method on your adapter (or a separate function) that matches the signature of an
http.HandlerFunc
(i.e.,func(w http.ResponseWriter, r *http.Request)
). In this function:- Validate the Request (Optional but recommended): Check the source, signature (if the external service provides one), or other headers to ensure the request is legitimate.
- Read and Parse the Request Body: The webhook payload will be in
r.Body
. Decode it (often JSON) into an appropriate Go struct. - Process the Notification: Perform the necessary actions based on the webhook's content. This might involve updating internal data, storing information, or publishing an internal event (see next section).
- Respond to the External Service: Send an appropriate HTTP response (usually
http.StatusOK
(200) if everything went well, or an error code otherwise). It's important to respond quickly to avoid timeouts on the external service's side.
Example handler function:
func (a *MyProviderAdapter) handleWebhookNotification(w http.ResponseWriter, r *http.Request) { ctx := r.Context() // Use the request's context slog.InfoContext(ctx, "Webhook notification received", "provider", a.GetID(), "path", r.URL.Path) // Optional: Validate the request (e.g., check a secret signature) var payload models.MyWebhookNotification // Your struct for the payload if err := json.NewDecoder(r.Body).Decode(&payload); err != nil { slog.ErrorContext(ctx, "Failed to decode webhook payload", "error", err) http.Error(w, "Invalid payload", http.StatusBadRequest) return } defer r.Body.Close() // ... Process the payload ... // For example, publish an internal event: // eventData := map[string]interface{}{"id": payload.ID, "status": payload.Status} // if err := a.notifier.Publish(ctx, fmt.Sprintf("%s.notification.received", a.GetID()), eventData); err != nil { // slog.ErrorContext(ctx, "Failed to publish webhook event", "error", err) // // Decide if this should result in an HTTP error for the webhook // } slog.InfoContext(ctx, "Webhook processed successfully", "payload_id", payload.ID) w.WriteHeader(http.StatusOK) fmt.Fprintln(w, "Notification received") }
The SLOP Server will automatically route incoming requests matching the declared paths to your handlers.
Publishing Events¶
Providers can publish events to the SLOP Server's internal Pub/Sub system. This allows for decoupling components: a provider can signal that an event has occurred without knowing who (if anyone) is listening or reacting to that event.
Key Steps:
-
Dependency on
Publisher
: Ensure your adapter receives an instance ofpubsub.Publisher
via its dependencies (injected by the factory and stored in a field of your adapter, e.g.,a.notifier
). -
Define Events (Optional but recommended): If your provider implements the
providers.EventInfoProvider
interface, you can declare the types of events it publishes via theGetEventDefinitions() []providers.EventDefinition
method. Eachproviders.EventDefinition
includes:ID
: A unique identifier for the event type (e.g.,"myprovider.entity.created"
).Description
: An explanation of what the event signifies.PayloadSchema
: A schema (often JSON Schema as amap[string]interface{}
) describing the expected structure of the event's payload.
Example
GetEventDefinitions
:// func (a *MyProviderAdapter) GetEventDefinitions() []providers.EventDefinition { // return []providers.EventDefinition{ // { // ID: fmt.Sprintf("%s.entity.created", a.GetID()), // Description: "Triggered when a new entity is created via MyProvider.", // PayloadSchema: map[string]interface{}{ // "type": "object", // "properties": map[string]interface{}{ // "entity_id": map[string]interface{}{"type": "string"}, // "details": map[string]interface{}{"type": "string"}, // }, // "required": []string{"entity_id"}, // }, // }, // } // }
-
Publish an Event: To publish an event, use the
Publish(ctx context.Context, eventID string, payload interface{}) error
method of thepubsub.Publisher
instance.eventID
: The unique identifier for the event (e.g.,"myprovider.entity.created"
).payload
: The data associated with the event. This can be any Go struct that can be serialized (usually to JSON), often amap[string]interface{}
or a specific struct.
Example of publishing an event:
// In a method of your adapter or service // func (a *MyProviderAdapter) createEntity(ctx context.Context, data MyEntityData) error { // // ... entity creation logic ... // entityID := "123" // details := "Some details" // // eventPayload := map[string]interface{}{ // "entity_id": entityID, // "details": details, // } // eventID := fmt.Sprintf("%s.entity.created", a.GetID()) // // if err := a.notifier.Publish(ctx, eventID, eventPayload); err != nil { // slog.ErrorContext(ctx, "Failed to publish entity creation event", "eventID", eventID, "error", err) // return err // Or handle the error differently // } // slog.InfoContext(ctx, "Entity creation event published", "eventID", eventID) // return nil // }
Published events can then be consumed by other parts of the system, such as configured event triggers that can launch jobs, call other tools, or notify users.
Registering and Implementing Jobs¶
Jobs are background tasks managed by the SLOP Server's central JobManager
. Your provider can define specific job types and provide the logic to execute them.
Key Steps:
-
Dependency on
JobManager
: Your adapter must have access to an instance ofjobmanager.Manager
, typically injected viaProviderDependencies
during its creation by the factory. -
Define Job Type: Choose a unique identifier for each job type your provider handles (e.g.,
"myprovider.sync_data"
). -
Implement Job Logic: The execution logic for a job is usually encapsulated in a function or method that takes a
context.Context
and the job parameters (amap[string]interface{}
). This function performs the actual work of the job. -
Register Job Handler: During your provider's initialization (e.g., in the adapter's
Initialize
method, or when the adapter is created), you must register a handler for each job type with theJobManager
. This is done by callingjobMgr.RegisterJobType(jobType string, handler jobmanager.JobHandlerFunc, options jobmanager.JobTypeOptions)
.jobType
: Your unique job type identifier.handler
: A function of typejobmanager.JobHandlerFunc
, which isfunc(ctx context.Context, jobID string, params map[string]interface{}) (map[string]interface{}, error)
. This function will be called by theJobManager
when a job of this type needs to be executed.options
: Configuration options for this job type (retry count, timeout, etc.).
Example job registration and handler:
// In your adapter.go or a dedicated *_job.go file const MyProviderSyncJobType = "myprovider.sync_data" // Handler function for the job func (a *MyProviderAdapter) handleSyncDataJob(ctx context.Context, jobID string, params map[string]interface{}) (map[string]interface{}, error) { slog.InfoContext(ctx, "Executing data sync job", "provider", a.GetID(), "jobID", jobID, "params", params) // Extract necessary parameters from `params` // userID, _ := params["user_id"].(string) // ... Data synchronization logic ... // err := a.syncUserData(ctx, userID) // if err != nil { // slog.ErrorContext(ctx, "Failed to sync data", "error", err) // return nil, err // The error will be stored by the JobManager // } result := map[string]interface{}{ "status": "success", "message": "Data synchronized.", } return result, nil // The result will be stored by the JobManager } // In your adapter's Initialize method (or at creation time) // func (a *MyProviderAdapter) Initialize(config map[string]interface{}) error { // // ... other initialization ... // // if a.jobMgr == nil { // return errors.New("JobManager not available for job type registration") // } // // jobOptions := jobmanager.JobTypeOptions{ // MaxRetries: 3, // Timeout: 5 * time.Minute, // } // if err := a.jobMgr.RegisterJobType(MyProviderSyncJobType, a.handleSyncDataJob, jobOptions); err != nil { // slog.Error("Failed to register sync job type", "error", err) // return fmt.Errorf("failed to register job type %s: %w", MyProviderSyncJobType, err) // } // slog.Info("Data sync job type registered successfully") // // // ... rest of initialization ... // return nil // }
-
Schedule Jobs: Once the job type is registered, jobs of this type can be scheduled from anywhere in the system (including by your own provider, by event handlers, or via the API) using
jobMgr.ScheduleJob(ctx context.Context, job jobmanager.Job) (*jobmanager.Job, error)
orjobMgr.ScheduleRecurringJob(...)
. Thejobmanager.Job
struct containsType
(your job type identifier) andParameters
(a map for the job's input data).The
unipile/adapter.go
example shows howscheduleAccountSyncJob
is used to schedule a recurring account synchronization job.
Jobs are a powerful way to perform long-running tasks, batch operations, or deferred actions without blocking the main request processing thread.
Managing Configuration¶
Each provider may require its own configuration (API keys, external service URLs, behavioral options, etc.). This configuration is managed via your adapter's Initialize(config map[string]interface{})
method.
Configuration Sources:
Configuration can come from two main sources, with a defined priority:
- Environment Variables: Often prefixed to avoid collisions (e.g.,
UNIPILE_API_KEY
,MYPROVIDER_BASE_URL
). These usually have the highest priority. - Central Configuration File (
config.yaml
or similar): The SLOP Server loads a global configuration file at startup. Provider-specific configurations are typically nested there under a key matching the provider's ID (e.g.,providers.myprovider.api_key
).
Reading Configuration in Initialize
:
In your adapter's Initialize
method, you receive a map[string]interface{}
representing your provider's configuration section extracted from the central file. You should:
- Access Values: Retrieve values from the map using expected keys (e.g.,
config["api_key"]
). - Check Type: Ensure the value is of the expected type (e.g.,
string
,bool
,int
). Use type assertions (e.g.,val, ok := config["api_key"].(string)
). - Fallback to Environment Variables: If a value is not found in the configuration map or is empty, try reading it from a corresponding environment variable (e.g.,
os.Getenv("MYPROVIDER_API_KEY")
). - Validate Configuration: Check that mandatory values are present and valid (e.g., a base URL must be a valid URL, an API key must not be empty).
- Store Configuration: Store the validated configuration values in your adapter struct's fields for later use by other provider methods.
Example configuration management in Initialize
(inspired by unipile/adapter.go
):
// func (a *MyProviderAdapter) Initialize(config map[string]interface{}) error {
// // ... (start of initialization)
// slog.InfoContext(ctx, "Initializing MyProviderAdapter with configuration")
// a.config = config // Store the raw config map if needed
// // --- API Key Configuration ---
// var apiKeyStr string
// apiKeyVal, ok := config["api_key"].(string)
// if ok && apiKeyVal != "" {
// apiKeyStr = apiKeyVal
// slog.InfoContext(ctx, "MyProvider API key loaded from config map.")
// } else {
// slog.WarnContext(ctx, "MyProvider API key not found or empty in config map, checking MYPROVIDER_API_KEY environment variable...")
// apiKeyStr = os.Getenv("MYPROVIDER_API_KEY")
// if apiKeyStr != "" {
// slog.InfoContext(ctx, "MyProvider API key loaded from MYPROVIDER_API_KEY environment variable.")
// } else {
// slog.ErrorContext(ctx, "MyProvider API key missing in config map and MYPROVIDER_API_KEY environment variable")
// return fmt.Errorf("MyProvider API key is required but not configured")
// }
// }
// a.apiKey = apiKeyStr // Store API key in adapter
// // --- Base URL Configuration ---
// var baseURLStr string
// baseURLVal, ok := config["base_url"].(string)
// // ... (similar logic for baseURL, with fallback to os.Getenv("MYPROVIDER_BASE_URL")) ...
// parsedBaseURL, err := url.ParseRequestURI(baseURLStr)
// if err != nil {
// slog.ErrorContext(ctx, "Failed to parse MyProvider base URL", "url", baseURLStr, "error", err)
// return fmt.Errorf("invalid MyProvider base URL rade_mark_sign%s rade_mark_sign: %w", baseURLStr, err)
// }
// a.baseURL = parsedBaseURL // Store parsed URL
// // ... (other configuration and client initialization) ...
// a.initialized = true
// slog.InfoContext(ctx, "MyProviderAdapter initialized successfully")
// return nil
// }
Using clear logs (slog
) to indicate where configuration is loaded from and to report configuration-related errors or warnings is recommended.
Implementing Storage¶
Some providers may need to persist data beyond what the CoreStore
(for main SLOP Server entities) or ResourceManager
(for shared data) offer.
Storage Options:
- Use the Main Database (PostgreSQL): This is the most common approach. Your provider can define its own tables in the SLOP Server's existing PostgreSQL database.
- External Storage Service: Less common, but a provider could interact with a dedicated storage service if needed.
Steps for PostgreSQL Storage:
-
Dependency on
pgxpool.Pool
: Your adapter must receive an instance of*pgxpool.Pool
viaProviderDependencies
(injected by the factory). This connection pool allows you to interact with the database. -
Define a Store Interface (Good Practice): Create an interface (e.g.,
MyProviderStoreInterface
) in your package that defines the database operations your provider needs (CRUD, specific queries).// type MyProviderStoreInterface interface { // SaveToken(ctx context.Context, userID string, token string) error // GetToken(ctx context.Context, userID string) (string, error) // InitializeDatabase(ctx context.Context) error // For schema creation // Close() // To close store resources if necessary // }
-
Implement the Store: Create a struct (e.g.,
pgMyProviderStore
) that implements this interface and holds the*pgxpool.Pool
.The// type pgMyProviderStore struct { // db *pgxpool.Pool // } // // func NewMyProviderStoreWithPool(pool *pgxpool.Pool) MyProviderStoreInterface { // return &pgMyProviderStore{db: pool} // } // // // Implement interface methods... // func (s *pgMyProviderStore) InitializeDatabase(ctx context.Context) error { /* ... SQL for CREATE TABLE IF NOT EXISTS ... */ }
unipile/pg_store.go
example shows aUnipilePgStore
implementation. -
Schema Initialization: If your provider defines its own tables, it must implement the
providers.SchemaInitializer
interface and itsInitializeSchema(ctx context.Context, pool *pgxpool.Pool) error
method. This method is called by the SLOP Server at startup and should contain the SQL statements (e.g.,CREATE TABLE IF NOT EXISTS ...
) to create or migrate necessary tables. It's common to call this schema initialization logic from your store's constructor or from the adapter'sInitialize
method after creating the store instance. Inunipile/adapter.go
,InitializeSchema
is implemented, and the actual database creation logic is called inNewUnipileAdapter
viastore.InitializeDatabase(initCtx)
. -
CRUD Operations: Implement your store interface's methods to interact with the database using the
*pgxpool.Pool
to execute SQL queries. -
Error and Transaction Handling: Properly handle database errors and use transactions (
pgx.Tx
) when multiple operations must be atomic.
Using the Store in the Adapter: Your provider-specific store instance is typically created in the adapter's factory (or in NewMyProviderAdapter
) and stored as a field in the adapter struct. Other adapter methods can then use this store for their persistence needs.
Testing Your Provider¶
Thoroughly testing your provider is crucial to ensure its reliability and correct functioning within the SLOP Server.
Types of Tests:
-
Unit Tests:
- Test individual functions and methods of your provider in isolation.
- Use mocks for external dependencies (API clients, database,
JobManager
,Publisher
). Go'stestify/mock
library is very useful for this. - Test configuration parsing logic, tool parameter validation, API request construction, response processing, etc.
- Example: Test a function that transforms data, or a method of your API client with a test HTTP server (
net/http/httptest
).
-
Integration Tests:
- Test your provider's interaction with actual SLOP Server components (or test versions of them) and with real external services (in a test/staging environment if possible).
- Adapter Testing: Instantiate your adapter, call
Initialize
with test configuration, then testGetProviderTools
andExecuteTool
.- For
ExecuteTool
, you can verify that the correct calls are made to your internal services or API clients (using mocks or test servers).
- For
- Webhook Testing: If your provider has webhooks, send test HTTP requests to these endpoints (by starting a test instance of the server or testing the handler directly) and verify correct processing.
- Job Testing: If your provider registers job types, you can test the job handler function directly. To test integration with the
JobManager
, you might need a testJobManager
or schedule a job and verify its execution (can be more complex). - Persistence Testing: If your provider uses a database store, test your store's CRUD operations against a test database (e.g., a temporary PostgreSQL instance or an in-memory database if compatible).
Testing Best Practices:
- Coverage: Aim for good code coverage with your tests.
- Isolation: Ensure tests are independent of each other.
- Reproducibility: Tests should yield the same results on every run.
- Clear Naming: Give descriptive names to your test files (e.g.,
adapter_test.go
,client_test.go
) and test functions. - Assertions: Use assertion libraries like
testify/assert
ortestify/require
for clear and concise checks. - Test Configuration: Use specific configuration files or environment variables for tests to avoid dependency on production configurations.
- Cleanup: Ensure tests clean up any resources they create (test database entries, temporary files).
The unipile
example does not directly provide tests in unipile_pkg_files.txt
, but the code structure (interfaces, separation of concerns) greatly facilitates writing unit and integration tests.
By following these guidelines, you will be able to effectively develop, integrate, and test new providers for the SLOP Server, extending its capabilities in a robust and maintainable manner.