open-notebook/docs/features/transformations.md

# Transformations

Transformations are a core feature of Open Notebook that provide a flexible and powerful way to generate new insights by applying customizable processing steps to your content. Inspired by the [Fabric framework](https://github.com/danielmiessler/fabric), transformations enable you to automatically distill, summarize, and enrich your research materials in meaningful ways.

## What are Transformations?

A **Transformation** is a customizable AI-powered process that modifies text input to produce structured, meaningful output. Whether you're summarizing articles, extracting key insights, generating reflective questions, or creating content outlines, transformations automate the processing of your research materials according to your specific needs.

Transformations work by:
- Taking your source content as input
- Applying a custom prompt template that defines the processing logic
- Using AI models to generate structured output
- Automatically creating new cards in your notebook with the results

## Core Components

### Transformation Elements

Each transformation consists of several key components:

- **Name**: Internal identifier for your reference
- **Title**: Displayed as the title of all cards created by this transformation
- **Description**: Helpful hint shown in the UI to explain the transformation's purpose
- **Prompt**: The actual AI prompt template that defines how content should be processed
- **Apply Default**: Whether this transformation should be suggested for all new sources

### Default Transformation Prompt

The system includes a configurable default transformation prompt that gets prepended to all transformations. This allows you to:
- Set consistent tone and style across all transformations
- Define global requirements or constraints
- Include instructions that prevent AI models from refusing certain tasks due to content policies

## Built-in Transformation Types

Open Notebook comes with several common transformation patterns that you can use immediately or customize:

### Content Analysis
- **Summarization**: Extract key points and main ideas from lengthy content
- **Insight Extraction**: Identify important insights, conclusions, and implications
- **Question Generation**: Create thoughtful questions for deeper reflection
- **Key Concepts**: Extract and define important terms and concepts

### Research Support
- **Literature Review**: Analyze academic papers and research content
- **Citation Extraction**: Pull out important quotes and references
- **Methodology Analysis**: Break down research methods and approaches
- **Data Insights**: Extract statistical findings and data points

### Creative Processing
- **Content Outlines**: Create structured outlines from unorganized content
- **Action Items**: Extract actionable tasks and next steps
- **Comparative Analysis**: Compare and contrast different perspectives
- **Trend Identification**: Spot patterns and emerging themes

## Custom Transformation Creation

### Creating Your Own Transformations

1. **Navigate to Transformations**: Go to the Transformations page in the UI
2. **Create New**: Click the "New Transformation" button
3. **Configure Settings**:
   - Enter a descriptive name for internal reference
   - Set a title that will appear on generated cards
   - Write a clear description explaining the transformation's purpose
   - Define your custom prompt template
   - Choose whether to apply by default to new sources

![New Transformation](/assets/new_transformation.png)

### Prompt Design Best Practices

When creating custom prompts, consider these guidelines:

**Structure Your Prompts**:
```
# ROLE
You are an expert researcher analyzing academic content.

# TASK
Extract the 5 most important insights from the following text.

# FORMAT
Present each insight as:
- **Insight**: [Brief description]
- **Evidence**: [Supporting details from text]
- **Implications**: [Why this matters]

# CONSTRAINTS
- Focus on actionable insights
- Avoid redundancy
- Cite specific examples from the text
```

**Use Template Variables**:
- Access source metadata with `{{ source.title }}`, `{{ source.url }}`
- Reference the current timestamp with `{{ current_time }}`
- Include custom data passed to the transformation

**Consider Output Format**:
- Use markdown for structured output
- Include headings for better organization
- Format lists and tables for readability

## Batch Processing Capabilities

### Applying Transformations at Scale

Transformations can be applied to multiple sources simultaneously:

1. **Source Selection**: Select multiple sources from your notebook
2. **Transformation Choice**: Choose which transformation to apply
3. **Batch Execution**: Process all selected sources with the same transformation
4. **Progress Tracking**: Monitor the processing status of each source

### Performance Considerations

- **Model Selection**: Choose appropriate models for your content type and complexity
- **Content Length**: Longer content may require more processing time and tokens
- **Concurrent Processing**: The system processes multiple transformations efficiently
- **Resource Management**: Monitor token usage and processing costs

## Transformation Management and Organization

### Organizing Your Transformations

**Categories and Tags**:
- Group related transformations by purpose
- Use descriptive names and clear descriptions
- Maintain a logical ordering for frequently used transformations

**Version Control**:
- Keep track of prompt changes over time
- Test modifications before applying to important content
- Maintain backup copies of successful transformation configurations

**Sharing and Collaboration**:
- Export transformation configurations for sharing
- Create standardized transformations for team use
- Document transformation purposes and best practices

## Integration with Other Features

### Notebook Integration

Transformations seamlessly integrate with your notebook workflow:

- **Automatic Card Creation**: Results appear as new cards in your notebook
- **Source Linking**: Transformed content maintains connections to original sources
- **Search Integration**: Transformation results are fully searchable
- **Note Connections**: Link transformation outputs to your personal notes

### Model Compatibility

Transformations work with various AI models:

- **OpenAI Models**: GPT-3.5, GPT-4, and other OpenAI offerings
- **Anthropic Models**: Claude variants with different capabilities
- **Local Models**: Self-hosted models for privacy and control
- **Specialized Models**: Domain-specific models for particular content types

### Workflow Integration

**Research Workflows**:
- Apply transformations as part of your research process
- Chain multiple transformations for complex analysis
- Use transformation results to guide further research

**Content Creation**:
- Transform research into actionable content
- Generate outlines and summaries for writing projects
- Extract quotes and citations for academic work

## Performance Considerations

### Optimization Strategies

**Model Selection**:
- Choose faster models for simple transformations
- Use more capable models for complex analysis
- Consider cost vs. quality trade-offs

**Prompt Optimization**:
- Write clear, specific prompts to reduce processing time
- Avoid overly complex instructions that may confuse models
- Test prompts with sample content before full deployment

**Content Preparation**:
- Pre-process content to remove unnecessary elements
- Break large documents into manageable chunks
- Ensure content is well-formatted for optimal results

### Monitoring and Troubleshooting

**Performance Metrics**:
- Track processing time for different transformation types
- Monitor token usage and associated costs
- Identify bottlenecks in your transformation pipeline

**Error Handling**:
- Implement retry mechanisms for failed transformations
- Log errors for debugging and improvement
- Provide fallback options for problematic content

## Best Practices and Use Cases

### Academic Research

**Literature Reviews**:
- Extract key findings from research papers
- Identify methodology patterns across studies
- Generate comparative analyses of different approaches

**Note-Taking Enhancement**:
- Transform raw notes into structured insights
- Generate questions for further investigation
- Create study guides from course materials

### Content Creation

**Blog Writing**:
- Transform research into blog post outlines
- Extract quotable insights and statistics
- Generate social media content from longer pieces

**Documentation**:
- Convert technical content into user-friendly guides
- Extract key procedures and best practices
- Create FAQ sections from support content

### Business Intelligence

**Market Research**:
- Analyze competitor content and strategies
- Extract trends and insights from industry reports
- Generate executive summaries from detailed analyses

**Process Improvement**:
- Transform feedback into actionable insights
- Identify patterns in customer communications
- Generate improvement recommendations from data

### Personal Knowledge Management

**Learning Enhancement**:
- Create study materials from educational content
- Generate practice questions from textbooks
- Extract key concepts for memorization

**Reflection and Planning**:
- Transform journal entries into insights
- Generate action items from meeting notes
- Create goal-setting materials from personal reflections

## Experimenting with Transformations

### Playground Environment

Use the Playground page to:
- Test different transformation prompts with sample content
- Compare results across different AI models
- Refine your transformations before applying to important content
- Experiment with new transformation ideas safely

### Iterative Improvement

**Testing Cycle**:
1. Create initial transformation prompt
2. Test with representative content samples
3. Analyze results and identify improvements
4. Refine prompt and test again
5. Deploy to production use

**Feedback Integration**:
- Collect feedback on transformation quality
- Iterate based on user needs and preferences
- Track transformation effectiveness over time

## Advanced Features

### Template Customization

**Dynamic Content**:
- Use conditional logic in prompt templates
- Adapt transformations based on source type
- Include context-sensitive instructions

**Variable Integration**:
- Access source metadata in transformations
- Include user preferences and settings
- Utilize historical transformation results

### Automation Workflows

**Scheduled Transformations**:
- Set up automatic processing for new content
- Create transformation pipelines for regular tasks
- Integrate with external content sources

**Conditional Processing**:
- Apply different transformations based on content type
- Use content analysis to guide transformation selection
- Implement quality checks and validation

## Troubleshooting Common Issues

### Transformation Failures

**Common Causes**:
- Malformed prompt templates
- Insufficient model capabilities
- Content formatting issues
- Token limit exceeded

**Solutions**:
- Validate prompt syntax before deployment
- Choose appropriate models for complexity
- Pre-process content for consistency
- Break large content into smaller chunks

### Quality Issues

**Poor Results**:
- Refine prompt specificity and clarity
- Provide more context and examples
- Adjust model selection for task complexity
- Test with different content types

**Inconsistent Output**:
- Standardize prompt formatting
- Include explicit output format requirements
- Use consistent terminology across prompts
- Implement validation checks

## Future Enhancements

The transformation system continues to evolve with planned features including:

- **Note Transformations**: Apply transformations to personal notes and annotations
- **Transformation Chains**: Link multiple transformations for complex workflows
- **Template Marketplace**: Share and discover transformation templates
- **Advanced Analytics**: Detailed metrics on transformation performance and usage
- **Integration APIs**: Connect transformations with external tools and services

## Conclusion

Transformations represent the heart of Open Notebook's intelligent content processing capabilities. By providing a flexible, customizable system for applying AI-powered analysis to your research materials, transformations enable you to extract maximum value from your content while maintaining control over the processing logic.

Whether you're conducting academic research, creating content, or managing personal knowledge, transformations can significantly enhance your productivity and insight generation. Start with the built-in transformation types, experiment with custom prompts in the playground, and gradually build a library of transformations tailored to your specific needs and workflows.

The sky truly is the limit when it comes to creating personalized, powerful workflows that bring out the most meaningful insights from your content.

<style scoped>
.custom-block.tip {
  border-color: var(--vp-c-brand);
  background-color: var(--vp-c-brand-dimm);
}

.custom-block.tip .custom-block-title {
  color: var(--vp-c-brand-darker);
}
</style>