CrucibleIR (CrucibleIR v0.3.0)
View SourceIntermediate Representation for the Crucible ML reliability ecosystem.
CrucibleIR provides shared data structures for defining ML reliability
experiments across the Crucible ecosystem. It includes definitions for
experiments, backends, datasets, reliability configurations, and more.
This package is data-only: structs, serialization, and structural validation. Execution logic and stage option validation belong in domain packages.
Main Components
CrucibleIR.Experiment- Top-level experiment definitionCrucibleIR.BackendRef- Reference to an LLM backendCrucibleIR.DatasetRef- Reference to a datasetCrucibleIR.StageDef- Processing stage definitionCrucibleIR.OutputSpec- Output specificationCrucibleIR.Reliability.Config- Container for reliability configurations
Backend IR
CrucibleIR.Backend.Prompt- Backend input contractCrucibleIR.Backend.Options- Backend generation optionsCrucibleIR.Backend.Completion- Backend output contractCrucibleIR.Backend.Capabilities- Backend capability discovery
Model Lifecycle (v0.2.0)
CrucibleIR.ModelRef- Reference to a registered modelCrucibleIR.ModelVersion- Specific model versionCrucibleIR.Training.Config- Training configurationCrucibleIR.Training.Run- Training execution recordCrucibleIR.Deployment.Config- Deployment configurationCrucibleIR.Deployment.Status- Deployment statusCrucibleIR.Feedback.Event- Feedback data pointCrucibleIR.Feedback.Config- Feedback collection configuration
Reliability Mechanisms
CrucibleIR.Reliability.Ensemble- Ensemble voting configurationCrucibleIR.Reliability.Hedging- Request hedging configurationCrucibleIR.Reliability.Stats- Statistical testing configurationCrucibleIR.Reliability.Fairness- Fairness checking configurationCrucibleIR.Reliability.Guardrail- Security guardrails configuration
Example
iex> alias CrucibleIR.{Experiment, BackendRef, StageDef}
iex> exp = CrucibleIR.new_experiment(
...> id: :my_exp,
...> backend: %BackendRef{id: :gpt4},
...> pipeline: [%StageDef{name: :inference}]
...> )
iex> exp.backend.id
:gpt4Summary
Functions
Creates a new experiment builder with the given ID.
Decodes JSON string to struct of given type.
Converts a map to struct of given type.
Creates a new experiment with the given attributes.
Encodes a struct to JSON string.
Returns true if struct is valid, false otherwise.
Validates a struct, returns {:ok, struct} or {:error, errors}.
Functions
Creates a new experiment builder with the given ID.
Delegates to CrucibleIR.Builder.experiment/1.
Examples
iex> exp = CrucibleIR.experiment(:test)
iex> exp.id
:testDecodes JSON string to struct of given type.
Delegates to CrucibleIR.Serialization.from_json/2.
Examples
iex> alias CrucibleIR.BackendRef
iex> json = ~s({"id":"gpt4","profile":"default"})
iex> {:ok, backend} = CrucibleIR.from_json(json, BackendRef)
iex> backend.id
:gpt4Converts a map to struct of given type.
Delegates to CrucibleIR.Serialization.from_map/2.
Examples
iex> alias CrucibleIR.BackendRef
iex> {:ok, backend} = CrucibleIR.from_map(%{"id" => "gpt4"}, BackendRef)
iex> backend.id
:gpt4@spec new_experiment(keyword()) :: CrucibleIR.Experiment.t()
Creates a new experiment with the given attributes.
This is a convenience function for creating CrucibleIR.Experiment structs.
Parameters
attrs- Keyword list of experiment attributes
Required Attributes
:id- Unique experiment identifier:backend- BackendRef struct:pipeline- List of StageDef structs
Optional Attributes
:description- Experiment description:owner- Experiment owner:tags- List of tags:metadata- Additional metadata map:dataset- DatasetRef struct:reliability- Reliability.Config struct:outputs- List of OutputSpec structs:created_at- Creation timestamp:updated_at- Update timestamp
Examples
iex> alias CrucibleIR.{BackendRef, StageDef}
iex> exp = CrucibleIR.new_experiment(
...> id: :test,
...> backend: %BackendRef{id: :gpt4},
...> pipeline: [%StageDef{name: :run}]
...> )
iex> exp.id
:test
iex> alias CrucibleIR.{BackendRef, StageDef, DatasetRef}
iex> alias CrucibleIR.Reliability.{Config, Stats}
iex> exp = CrucibleIR.new_experiment(
...> id: :exp1,
...> backend: %BackendRef{id: :gpt4},
...> pipeline: [%StageDef{name: :run}],
...> dataset: %DatasetRef{name: :mmlu},
...> reliability: %Config{stats: %Stats{alpha: 0.01}}
...> )
iex> exp.reliability.stats.alpha
0.01Encodes a struct to JSON string.
Delegates to CrucibleIR.Serialization.to_json/1.
Examples
iex> alias CrucibleIR.BackendRef
iex> json = CrucibleIR.to_json(%BackendRef{id: :gpt4})
iex> is_binary(json)
trueReturns true if struct is valid, false otherwise.
Delegates to CrucibleIR.Validation.valid?/1.
Examples
iex> alias CrucibleIR.BackendRef
iex> CrucibleIR.valid?(%BackendRef{id: :gpt4})
trueValidates a struct, returns {:ok, struct} or {:error, errors}.
Delegates to CrucibleIR.Validation.validate/1.
Examples
iex> alias CrucibleIR.{Experiment, BackendRef, StageDef}
iex> exp = %Experiment{
...> id: :test,
...> backend: %BackendRef{id: :gpt4},
...> pipeline: [%StageDef{name: :run}]
...> }
iex> {:ok, _} = CrucibleIR.validate(exp)