1
00:00:00,050 --> 00:00:06,110
Case study, structured AI model versioning, enhancing fraud detection and governance at fin edge in

2
00:00:06,110 --> 00:00:07,160
2021.

3
00:00:07,190 --> 00:00:13,100
A fintech company named Fin Edge faced a critical challenge with their machine learning model used for

4
00:00:13,100 --> 00:00:15,230
detecting fraudulent transactions.

5
00:00:16,130 --> 00:00:22,070
Despite its initial success, the model's performance started deteriorating, leading to an increase

6
00:00:22,070 --> 00:00:24,440
in undetected fraudulent activities.

7
00:00:24,980 --> 00:00:30,800
This decline prompted the company's AI governance team to implement a structured approach to model versioning

8
00:00:30,800 --> 00:00:36,890
and updates, ensuring the model stayed effective, ethical, and aligned with Fin Edge's goals.

9
00:00:38,060 --> 00:00:43,850
Fin Edge's AI governance team, led by Doctor Alice Kang, focused on establishing a clear versioning

10
00:00:43,850 --> 00:00:45,830
scheme for their fraud detection model.

11
00:00:46,340 --> 00:00:50,750
Each version of the model was uniquely identifiable using semantic versioning.

12
00:00:51,020 --> 00:00:55,640
For example, the initial model was labeled as version 1.0.0.

13
00:00:56,090 --> 00:00:59,060
Subsequent iterations followed a consistent pattern.

14
00:00:59,090 --> 00:01:02,090
A major version change for significant updates.

15
00:01:02,120 --> 00:01:06,950
A minor version for incremental improvements and a patch version for bug fixes.

16
00:01:06,950 --> 00:01:11,360
This approach provided clarity and helped stakeholders track the model's evolution.

17
00:01:12,470 --> 00:01:16,730
One of the first steps was to document the changes and improvements in each version.

18
00:01:17,180 --> 00:01:22,940
The documentation included details about the model's architecture, training data, hyperparameters,

19
00:01:22,940 --> 00:01:24,560
and performance metrics.

20
00:01:25,160 --> 00:01:28,250
Doctor Kang emphasized the importance of this practice.

21
00:01:28,580 --> 00:01:34,610
She argued that comprehensive documentation not only facilitated informed decision making, but also

22
00:01:34,610 --> 00:01:37,580
ensured compliance with regulatory requirements.

23
00:01:38,450 --> 00:01:43,730
This practice raised the question how can maintaining thorough documentation aid in the compliance and

24
00:01:43,730 --> 00:01:45,650
governance of AI models?

25
00:01:46,850 --> 00:01:53,030
The team also integrated automated tools like MLflow and DVC for managing model artifacts and tracking

26
00:01:53,030 --> 00:01:53,960
experiments.

27
00:01:54,710 --> 00:02:00,290
These tools provided functionalities for versioning models and managing model artifacts, integrating

28
00:02:00,290 --> 00:02:02,840
seamlessly with fin Edge's continuous integration.

29
00:02:02,840 --> 00:02:04,980
Continuous deployment pipelines.

30
00:02:05,700 --> 00:02:11,040
Automation played a crucial role by reducing human error and ensuring reproducibility, thus speeding

31
00:02:11,040 --> 00:02:12,540
up the deployment process.

32
00:02:13,350 --> 00:02:19,470
This led to another important question what are the benefits and potential drawbacks of using automated

33
00:02:19,500 --> 00:02:22,080
tools for model versioning and updates?

34
00:02:23,700 --> 00:02:29,220
As weeks passed, the team noticed a further decline in the model's performance due to data drift.

35
00:02:29,640 --> 00:02:36,570
Monitoring tools tracked key performance indicators such as accuracy, precision recall, and F1 score.

36
00:02:36,870 --> 00:02:42,210
Anomaly detection mechanisms alerted them to unusual patterns, prompting immediate investigation.

37
00:02:42,510 --> 00:02:47,670
This situation highlighted the necessity of regular monitoring and evaluation to identify the need for

38
00:02:47,670 --> 00:02:48,450
updates.

39
00:02:49,410 --> 00:02:55,230
Doctor Kang asked her team how can continuous monitoring and anomaly detection improve the performance

40
00:02:55,230 --> 00:02:57,330
and reliability of AI models?

41
00:02:58,710 --> 00:03:04,170
When the decision was made to update the model, the team conducted thorough testing and validation.

42
00:03:04,740 --> 00:03:07,820
They used offline evaluation with historical data.

43
00:03:07,850 --> 00:03:14,060
A b testing with live traffic and shadow deployment to ensure the new model did not negatively impact

44
00:03:14,090 --> 00:03:15,470
real world outcomes.

45
00:03:15,830 --> 00:03:21,530
Involving a diverse team in the testing phase helped uncover potential biases and blind spots.

46
00:03:22,040 --> 00:03:27,530
This raised the question why is it important to involve a diverse team in the testing and validation

47
00:03:27,530 --> 00:03:29,090
phase of model updates?

48
00:03:30,710 --> 00:03:34,370
Ethical considerations were paramount during the update process.

49
00:03:34,610 --> 00:03:38,930
The team regularly audited the training data for representativeness and fairness.

50
00:03:39,110 --> 00:03:45,110
Techniques like adversarial debiasing were employed to mitigate biases and promote equitable outcomes.

51
00:03:45,470 --> 00:03:51,950
This vigilance in ensuring ethical considerations led Doctor Kang to ask, what are the potential consequences

52
00:03:51,950 --> 00:03:55,340
of failing to address biases in AI models?

53
00:03:56,780 --> 00:04:01,340
Transparency and communication were integral to implementing model updates.

54
00:04:01,670 --> 00:04:06,770
Funage ensured that all stakeholders, including end users, were informed about significant changes

55
00:04:06,770 --> 00:04:10,210
to the model, their rationale and expected impacts.

56
00:04:10,450 --> 00:04:15,250
This transparency built trust and allowed users to provide valuable feedback.

57
00:04:16,120 --> 00:04:21,880
Doctor Kang posed the question how can transparency and open communication enhance trust and collaboration

58
00:04:21,880 --> 00:04:23,200
among stakeholders?

59
00:04:24,550 --> 00:04:30,370
The alignment of model updates with broader organizational strategies and goals was another critical

60
00:04:30,370 --> 00:04:31,390
consideration.

61
00:04:31,810 --> 00:04:37,930
The AI governance team regularly reviewed the alignment between the AI models and Finn Edge's objectives.

62
00:04:38,410 --> 00:04:43,930
This practice ensured that the models contributed to the company's mission and adapted to shifts in

63
00:04:43,930 --> 00:04:46,480
strategy or external conditions.

64
00:04:46,840 --> 00:04:53,080
This led to the question how does aligning AI model updates with organizational goals facilitate the

65
00:04:53,080 --> 00:04:55,060
strategic direction of a company?

66
00:04:56,290 --> 00:05:01,300
Finally, the team developed a robust rollback strategy to manage model updates.

67
00:05:01,330 --> 00:05:07,150
This plan allowed Finn Edge to revert to a previous stable version if an update led to unforeseen issues

68
00:05:07,150 --> 00:05:08,140
in production.

69
00:05:08,230 --> 00:05:13,990
Integrating this strategy into the CI CD pipeline ensured that rollbacks could be executed quickly and

70
00:05:13,990 --> 00:05:16,420
efficiently, minimizing disruption.

71
00:05:16,840 --> 00:05:22,180
This prompted Doctor Kang to ask, why is it essential to have a well defined rollback strategy in place

72
00:05:22,180 --> 00:05:23,830
for AI model updates?

73
00:05:26,020 --> 00:05:31,570
To conclude, the case study of Fin edge provides a practical example of the importance of structured

74
00:05:31,570 --> 00:05:35,050
model versioning and updates in AI system management.

75
00:05:35,890 --> 00:05:41,920
Maintaining thorough documentation helped ensure compliance and facilitated informed decision making.

76
00:05:42,670 --> 00:05:49,150
The use of automated tools like MLflow and DVC for versioning and management significantly reduced human

77
00:05:49,150 --> 00:05:51,220
error and improved reproducibility.

78
00:05:51,220 --> 00:05:56,830
Regular monitoring and evaluation allowed the team to identify performance degradation and data drift

79
00:05:56,830 --> 00:06:04,450
promptly, ensuring timely updates involving a diverse team in the testing and validation phase, uncovered

80
00:06:04,450 --> 00:06:09,310
biases and blind spots, improving the model's performance and fairness.

81
00:06:09,790 --> 00:06:15,010
Ethical considerations remained at the forefront, and techniques like adversarial debiasing helped

82
00:06:15,010 --> 00:06:21,430
mitigate biases, promoting equitable outcomes, transparency and open communication with stakeholders.

83
00:06:21,430 --> 00:06:24,400
Built trust and facilitated valuable feedback.

84
00:06:24,400 --> 00:06:26,020
Enhancing collaboration.

85
00:06:27,250 --> 00:06:32,860
Aligning model updates with organizational goals ensured that the AI systems contributed to Fin Edge's

86
00:06:32,860 --> 00:06:36,910
mission and adapted to shifts in strategy or external conditions.

87
00:06:37,420 --> 00:06:42,850
Finally, having a robust rollback strategy in place allowed the team to quickly revert to a previous

88
00:06:42,850 --> 00:06:49,060
stable version in case of unforeseen issues, minimizing disruption and maintaining service continuity.

89
00:06:51,460 --> 00:06:57,190
In essence, the case study underscores the necessity of adopting best practices for model versioning

90
00:06:57,190 --> 00:06:58,180
and updates.

91
00:06:58,930 --> 00:07:04,330
By adhering to these practices, AI governance professionals can maintain the effectiveness, fairness,

92
00:07:04,330 --> 00:07:10,300
and alignment of AI models, ensuring that deployed AI systems continue to deliver value while adhering

93
00:07:10,330 --> 00:07:12,700
to ethical and regulatory standards.