Skill validation

Skill validation is a crucial aspect of ensuring the quality and reliability of skills created in Root Signals. Validators consist of an optional guard value and an evaluator. By attaching validators to a skill, you can continuously monitor its performance and block outputs that do not meet the required standards.

Validators are a powerful tool to measure, control and ensure the quality of the skills you create in Root Signals. They allow you to confidently deploy skills knowing they will meet your quality bar.

Setting Guard Values

If you choose a validator for a skill but do not set a threshold, then validation results are generated after the skill execution in an asynchronous fashion. Validation results for skill executions are always accessible through the monitoring view.

When attaching a validator to a skill, you have the option to set a guard value. The guard value is a threshold that determines the minimum acceptable performance level for the skill. If the validator returns a number below the threshold, the validator is considered to not pass and the LLM response is not returned to the user. For example, if you set a guard value of 0.5 for the Quality of Professional Writing validator, the skill's output must meet the criteria of being professional, correct, concise, and precise at least to a level of 0.5.

Guard values ensure that only high-quality responses are returned to the user.

Monitoring Validation Results

Root Signals provides a monitoring view where you can track the validation results for each skill execution. You can click on an individual execution to see more details, including the specific validation results for that execution.

This monitoring capability allows you to keep a close eye on the performance of your skills and identify any issues or areas for improvement.

Conclusion

Skill validation is an essential feature in Root Signals that enables you to ensure the quality, accuracy, and appropriateness of your skills. By utilizing the built-in validators, setting guard values, monitoring validation results, and testing skills against different models, you can create robust and reliable skills that meet your organization's standards. Validators provide an objective measure of skill performance, giving you confidence in the skills you deploy and share within your organization.