Building Machine Learning Powered Applications

Table of contents

1. Preface
   1. The Goal of Using Machine Learning Powered Applications
      1. Use ML to Build Practical Applications
      2. Additional Resources
   2. Practical ML
      1. What This Book Covers
      2. Prerequisites
      3. Our Case Study: ML–Assisted Writing
      4. The ML Process
   3. Conventions Used in This Book
   4. Using Code Examples
   5. O’Reilly Online Learning
   6. How to Contact Us
   7. Acknowledgments
2. I. Find the Correct ML Approach
3. 1. From Product Goal to ML Framing
   1. Estimate What Is Possible
      1. Models
      2. Data
   2. Framing the ML Editor
      1. Trying to Do It All with ML: An End-to-End Framework
      2. The Simplest Approach: Being the Algorithm
      3. Middle Ground: Learning from Our Experience
   3. Monica Rogati: How to Choose and Prioritize ML Projects
   4. Conclusion
4. 2. Create a Plan
   1. Measuring Success
      1. Business Performance
      2. Model Performance
      3. Freshness and Distribution Shift
      4. Speed
   2. Estimate Scope and Challenges
      1. Leverage Domain Expertise
      2. Stand on the Shoulders of Giants
   3. ML Editor Planning
      1. Initial Plan for an Editor
      2. Always Start with a Simple Model
   4. To Make Regular Progress: Start Simple
      1. Start with a Simple Pipeline
      2. Pipeline for the ML Editor
   5. Conclusion
5. II. Build a Working Pipeline
6. 3. Build Your First End-to-End Pipeline
   1. The Simplest Scaffolding
   2. Prototype of an ML Editor
      1. Parse and Clean Data
      2. Tokenizing Text
      3. Generating Features
   3. Test Your Workflow
      1. User Experience
      2. Modeling Results
   4. ML Editor Prototype Evaluation
      1. Model
      2. User Experience
   5. Conclusion
7. 4. Acquire an Initial Dataset
   1. Iterate on Datasets
      1. Do Data Science
   2. Explore Your First Dataset
      1. Be Efficient, Start Small
      2. Insights Versus Products
      3. A Data Quality Rubric
   3. Label to Find Data Trends
      1. Summary Statistics
      2. Explore and Label Efficiently
      3. Be the Algorithm
      4. Data Trends
   4. Let Data Inform Features and Models
      1. Build Features Out of Patterns
      2. ML Editor Features
   5. Robert Munro: How Do You Find, Label, and Leverage Data?
   6. Conclusion
8. III. Iterate on Models
9. 5. Train and Evaluate Your Model
   1. The Simplest Appropriate Model
      1. Simple Models
      2. From Patterns to Models
      3. Split Your Dataset
      4. ML Editor Data Split
      5. Judge Performance
   2. Evaluate Your Model: Look Beyond Accuracy
      1. Contrast Data and Predictions
      2. Confusion Matrix
      3. ROC Curve
      4. Calibration Curve
      5. Dimensionality Reduction for Errors
      6. The Top-k Method
      7. Other Models
   3. Evaluate Feature Importance
      1. Directly from a Classifier
      2. Black-Box Explainers
   4. Conclusion
10. 6. Debug Your ML Problems
    1. Software Best Practices
       1. ML-Specific Best Practices
    2. Debug Wiring: Visualizing and Testing
       1. Start with One Example
       2. Test Your ML Code
    3. Debug Training: Make Your Model Learn
       1. Task Difficulty
       2. Optimization Problems
    4. Debug Generalization: Make Your Model Useful
       1. Data Leakage
       2. Overfitting
       3. Consider the Task at Hand
    5. Conclusion
11. 7. Using Classifiers for Writing Recommendations
    1. Extracting Recommendations from Models
       1. What Can We Achieve Without a Model?
       2. Extracting Global Feature Importance
       3. Using a Model’s Score
       4. Extracting Local Feature Importance
    2. Comparing Models
       1. Version 1: The Report Card
       2. Version 2: More Powerful, More Unclear
       3. Version 3: Understandable Recommendations
    3. Generating Editing Recommendations
    4. Conclusion
12. IV. Deploy and Monitor
13. 8. Considerations When Deploying Models
    1. Data Concerns
       1. Data Ownership
       2. Data Bias
       3. Systemic Bias
    2. Modeling Concerns
       1. Feedback Loops
       2. Inclusive Model Performance
       3. Considering Context
       4. Adversaries
       5. Abuse Concerns and Dual-Use
    3. Chris Harland: Shipping Experiments
    4. Conclusion
14. 9. Choose Your Deployment Option
    1. Server-Side Deployment
       1. Streaming Application or API
       2. Batch Predictions
    2. Client-Side Deployment
       1. On Device
       2. Browser Side
    3. Federated Learning: A Hybrid Approach
    4. Conclusion
15. 10. Build Safeguards for Models
    1. Engineer Around Failures
       1. Input and Output Checks
       2. Model Failure Fallbacks
    2. Engineer for Performance
       1. Scale to Multiple Users
       2. Model and Data Life Cycle Management
       3. Data Processing and DAGs
    3. Ask for Feedback
    4. Chris Moody: Empowering Data Scientists to Deploy Models
    5. Conclusion
16. 11. Monitor and Update Models
    1. Monitoring Saves Lives
       1. Monitoring to Inform Refresh Rate
       2. Monitor to Detect Abuse
    2. Choose What to Monitor
       1. Performance Metrics
       2. Business Metrics
    3. CI/CD for ML
       1. A/B Testing and Experimentation
       2. Other Approaches
    4. Conclusion
17. Index