Can AI outperform humans? For years, this question remained unanswered—until 2016, when AlphaGo made history by defeating Lee Sedol. AlphaGo, an AI program developed by Google DeepMind, utilized deep learning and reinforcement learning to master the complex game of Go. In March 2016, it triumphed over South Korean 9-dan professional Go player Lee Sedol, winning 4 games to 1, demonstrating the true potential of AI to the world.
Go is known for its immense complexity, with an estimated 10170 possible outcomes—more than the number of atoms in the universe. AlphaGo, leveraging vast computational power, learned and refined its strategy in real time. However, the Go board is a highly controlled environment, with all variables known and finite. In such a setting, AlphaGo’s victory was almost inevitable.
But the real world is far more complex than a Go board. AlphaGo was trained on high-quality, publicly available data from world-class players, optimizing its performance within a structured environment. In contrast, the challenges in the real world are much more unpredictable.
Take the complexity of a manufacturing floor as an example. Variables like temperature, humidity, noise, equipment wear, and scheduling all interact, influencing machine performance and even the timing of part replacements. Additionally, variations in worker skills and changing conditions add to the dynamic nature of the production environment. This complexity is far beyond what AlphaGo was designed to handle.
What if AI could handle the unpredictable environments of industrial sites? Unlike AlphaGo, which operates in a structured game setting, AI trained on real-world data can learn to manage a broad range of variables. In industry, the ongoing challenge is maximizing efficiency and productivity. However, applying AI directly to functioning processes carries risks.
Industrial sites are intricate systems where countless variables interact. If AI models aren’t rigorously validated in real-world scenarios, they could fail unpredictably, potentially interrupting processes or compromising product quality. That’s why a cautious, methodical approach is crucial when implementing new technologies like AI. Testing AI in real-world environments without proper simulation is impractical and risky.
AI-powered digital twins simulate real-world complexities, allowing companies to test AI models in a risk-free environment before live deployment. MakinaRocks has developed its own multi-step simulation dynamics model, creating a digital twin that replicates industrial environments, including processes, equipment, products, and operational conditions. This enables rapid validation and optimization of AI models for tasks like prediction, optimization, and predictive maintenance, ensuring safer innovation without disrupting current operations.
Illustration of MakinaRocks’ AI-powered digital twin
In the industrial world, AI-powered digital twins are transforming productivity, operational efficiency, and risk management. While traditional equipment simulators provide valuable insights, they often fall short of capturing real-time changes in process conditions. MakinaRocks’ digital twin overcomes this limitation by offering real-time monitoring of both equipment and process conditions, as well as external factors and operator behavior. This creates a highly accurate and dynamic simulation.
Unlike basic simulators that predict only the next step (single step), MakinaRocks’ digital twin can forecast mid- to long-term (multi-step) state changes, offering a deeper understanding of future developments. By incorporating observation variables, internal and external conditions, and control inputs from real industrial data, the digital twin enables testing of multiple control scenarios. Combined with advanced AI models, this technology effectively addresses complex industrial challenges such as preventing critical failures, optimizing control systems, and improving production schedules.
How are digital twins transforming industries? Let’s look at a practical example in 🔗 energy efficiency for an electric vehicle's air conditioning system. By providing current state data, target temperature, and desired energy efficiency, a reinforcement learning agent calculates control values for the system’s actuators. These values are then fed into the AI-powered digital twin, which simulates the predicted temperature and energy efficiency. The results are compared to the target, and the agent is rewarded based on how closely the goals are met. Through continuous feedback, the agent optimizes control values, ultimately achieving the desired temperature and efficiency.
Beyond electric vehicle air conditioning, MakinaRocks has applied this approach across multiple domains, including 🔗 tuning parameters of industrial machines, 🔗 optimizing temperatures in steel furnaces, and automating waste incinerator operations. In fact, more than 5,000 AI models deployed by MakinaRocks are being validated and used in real-world digital twins, providing precise, AI-driven insights that mirror actual conditions.
Curious about how to leverage AI for real business impact in the complex, unpredictable industrial landscape? Click the banner below to contact us. We’re here to help you turn possibilities into reality.
At MakinaRocks, we believe AI can make the world a better place. Its innovations will extend far beyond office productivity and digital environments. We see AI creating unprecedented advancements in the real world, where we breathe, move, and interact.
In the vast expanse of the real world, we harness AI to intelligentize industrial sites. With endless streams of industrial data from countless machines, we solve complex challenges and drive real business impact. The industrial floor is constantly evolving, and the challenges are becoming tougher. We break new ground in this complex and unpredictable environment with industry-specific AI.
What lies ahead for real-world AI? At its core, AI is a technology designed for people. It rapidly and accurately solves real-world problems while also uncovering new solutions that humans might not envision. Real-world AI is about harmonizing people and technology to build a better world. We’re creating a future where the industrial workspace is intelligent, allowing people to focus on what they do best.
General-purpose AI like ChatGPT enhance office productivity by translating, writing, coding, and more in digital environments. Fashion e-commerce platforms and content platforms offer personalized recommendations. It’s almost impossible to navigate the digital world without AI. But what about the economic implications in the real world? According to the World Bank, the digital economy contributes over 15% of global gross domestic product (GDP). While the digital economy is rapidly expanding, the physical world still constitutes the majority of the global economy. We believe AI will have an even greater impact when applied to these vast markets. AI has the potential to revolutionize industries such as automotive, robotics, semiconductors, batteries, energy, logistics, and more.
The value of AI in real-world industries is immense. For instance, in manufacturing, AI can enhance demand forecasting, enabling logistics to optimize inventory management and delivery planning. This leads to faster delivery times and cost reductions. Retailers can also optimize inventory and plan sales promotions based on AI-driven demand forecasts. As AI applications in manufacturing increase, IT companies will develop new solutions and services to support them, driving further R&D investment and technological advancements.
The potential of real-world AI is just the tip of the iceberg. AI and data are intrinsically linked; AI needs data to function, and data determines AI’s effectiveness. In the digital world, vast amounts of online data can be easily collected and analyzed at relatively low costs. But what about the real world?
Look around your surroundings. We generate massive amounts of data daily. Home appliances, vehicles, daily routes, and objects we use all produce data. AI can analyze this data to make our lives more efficient, solve unpredictable problems, and unlock new possibilities.
In industrial environments, machines continuously generate data such as vibrations, temperatures, and sounds. A well-designed AI system comprehends the physical laws and context of this data. It can recognize real-time events on the industrial floor and control physical elements like equipment and robots. AI is redefining automation from rule-based tasks to intelligent operations. However, obtaining high-quality, consistent data remains a significant challenge. There is a wealth of untapped data in the real world, and harnessing this data can unleash AI’s full potential.
Interested in exploring the new horizons of real-world AI, with its vast markets and immense potential? Stay tuned for the next installment of our series, where we’ll showcase how we’re making real-world AI a reality with enterprise customers across various industries, including automotive, semiconductor, battery, chemical, defense, retail, and utilities. In the meantime, if you’d like to discuss real-world AI for your organization, click the banner below to reach us. We’re here to help.
Hello, we are Seokgi Kim and Jongha Jang, AI engineers at MakinaRocks! At MakinaRocks, we are revolutionizing industrial sites by identifying problems that can be solved with AI, defining these problems, and training AI models to address them. Throughout numerous AI projects, we have recognized the critical importance of experiment management. This insight was particularly highlighted while working on an AI project for component prediction with an energy company. Our experiences led us to develop and integrate a robust experiment management function into our AI platform, Runway. In this post, we’ll share how we systematized our experiment management approach and our learnings from various AI projects.
The core of any AI project is the performance of the models we develop. Improving model performance requires a structured approach to conducting and comparing experiments. When developing models, we often test various methods by individual team members. It's crucial that these experiments are conducted under consistent conditions to allow for accurate performance comparisons between methodologies.
For instance, if a model developed using Methodology A with extensive data outperforms a model using Methodology B with less data, we can't conclusively say that Methodology A is superior without considering the data volume differences. Effective experiment management helps us control such variables and make more valid comparisons.
Moreover, team collaboration and client interactions often lead to changes in training data, complicating the comparison of models from different experiments. For example, if a client requests a last-minute change to the training data, it's essential to have detailed records of the previous data and its processing. Without this, comparing past and current experiments' performance becomes challenging.
The potential for confusion exists not only between different team members but also within your own experiments. As you work on a project and collaborate with clients, changes to the training data are inevitable, making it challenging to accurately compare the model performance of different experiments. For instance, if you conducted an experiment a month ago and the client requests a last-minute change to the training data, it's crucial to have detailed records of the previous data and its processing. Without these records, or if reproducing the old data is too difficult, comparing the performance of new experiments with old ones becomes nearly impossible.
An experiment management system is indispensable for ensuring the success of AI projects. Here are key reasons why it’s necessary:
To address these needs, we established a set of principles for experiment management throughout the project and devised a systematic approach to manage them efficiently. Here’s how we created our experiment management system.
To efficiently manage experiments in AI model development projects, we have organized our experiment management into three main areas:
With this structured approach to data versioning, it’s straightforward to compare the performance of two experiments if the data versions used are, for example, v0.X.Y and v0.X.Z. Since the test sets are identical, any differences in performance can be attributed to changes in the model or methodology rather than variations in the data.
Managing Source Code Versions in Model Development
In a model development project, maintaining a record of the different attempts to improve model performance is crucial. With continuous trial and error, the direction of the project often changes. Consequently, the source code is in a constant state of flux. Without proper source code management, reproducing experiments can become challenging, and team members might find themselves rewriting code when they attempt to replicate methodologies.
To address these challenges, we focused on maintaining a one-to-one correspondence between experiments and source code versions. We managed our source code using Git Tags, and here’s how we did it:
💡 Experiment Process for Source Code Version Control
1.Pull Request Merge
When a team member wants to try a new experiment, they write a pull request (PR) with the necessary code. The rest of the team reviews the PR, and upon approval, it is merged into the main branch.
2.Git Tag Release
After merging the branch, a new release is created. We create a Git tag for this release, including a brief description in the release note. This description outlines the experiments that have been updated or the new features added to enhance the experiment’s visibility compared to the previous version.
3. Logging Git Tags When Running Experiments
When running an experiment, it’s important to log the Git tag in MLflow. This practice makes it easy to track which source code version each experiment was run on. In our project, we log the source code version of each experiment as an MLflow tag named src_version in the MLflow run.
By following this process, each source code version is systematically generated and released as a Git tag.
Source code version v0.5.1 release note
On the MLflow screen, the source code version is logged as an MLflow Tag. This allows users to check the “Columns” section to quickly see which source code version each experiment was run on.
TAME-Data-v0.4.0 part of Experiment display
When using MLflow to manage experiments, the Git Commit ID is automatically logged with each experiment. However, there are distinct advantages to using Git Tags to specify and log source code versions.
💡 Benefits of Using Git Tags for Source Code Versioning in AI Projects
1. Easily Identify Source Code Verisons
When viewing multiple experiments in the Experiments list, Git Tags allow you to see at a glance which experiments were run with the same source code version.
2. Understand the Temporal Order of Experiments
With Git Commit IDs, determining the order of experiments can be challenging since the commit hashes are random strings. Although you can compare commits directly, it’s not straightforward. Git Tags, on the other hand, provide version numbers that make it easy to understand the sequence of experiments. This is particularly advantageous for projects in the model development phase, where the temporal order of experiments is crucial.
3. Document Source Code Versions with Release Notes
Using Git Tags comes with the natural benefit of release notes, which document each source code version. As experiments accumulate, having detailed release notes helps in tracking methodologies and understanding the progression of the project. This documentation is invaluable for both code management and project traceability, offering insights into the project timeline and the context of each version.
As mentioned earlier, we use an open-source tool called MLflow to manage our experiments effectively, especially in an AI project with dozens to tens of thousands of experiments. Here’s an explanation of how MLflow is utilized to manage these experiments.
The key to recording experiments in MLflow is ensuring comprehensive and clear descriptions. Below is a detailed example.
First, MLflow Experiments were created for each data version to facilitate performance comparisons. In the Description section of each MLflow Experiment for a data version, a brief description of the data version is provided.
Below is one of the example screens for MLflow Experiment.
Example MLflow screen for Data Version v0.4.0
In the Description section of the MLflow screen for Data Version v0.4.0, the following details are included:
Will this help us manage experiments perfectly if we write detailed descriptions? As the model evolves and multiple data versions emerge, finding the right experiment can become time-consuming. To address this, experiments are recorded in separate spaces based on their purpose, making it easier to locate specific experiments later.
To organize experiments according to their purpose, the following spaces were created:
Separated experimental spaces based on the purpose of the experiment
[Type 1] Project Progress Leaderboard
This space records experiments to show the overall progress of the project. As the data version changes, experiments that perform well or are significant for each version are recorded here, providing a comprehensive overview of the project’s development.
[Type 2] Data Version Experiment Space
This is the main experiment space where all experiments are recorded according to the data version. With a focus on comparing model performance across experiments, an experiment space is created for each data version. Detailed information about the data used to train the model, model parameter values, and other relevant details are recorded to ensure the reproducibility of experiments.
[Type 3] Experiment Space for Test Logging
The test logging space serves as a notepad-like environment. It is used to verify whether experiment recording works as intended and is reproducible during source code development. This space allows for temporary, recognizable names (like the branch name being worked on). Once test logging confirms that an experiment is effective, the details are logged back into the main experiment space, the “[Type 2] Data Version Experiment Space.”
[Type 4] Hyperparameter Tuning Experiment Space
All hyperparameter tuning experiments for the main experiments are logged in this separate space. This segregation ensures that the main experiment space remains focused on comparing the performance of different methodologies or models. If hyperparameter tuning experiments were logged in the main space, it would complicate performance comparisons. Once a good hyperparameter combination is found, it is applied and logged in the main experiment space. Logging hyperparameter exploration helps in analyzing trends, which can narrow down the search space and increase the likelihood of finding better-performing hyperparameters. For example, here's a graph that analyzes the trend of depth, one of the hyperparameters in the CatBoost model.
Trend analysis graph of the depth hyperparameter in CatBoost models
When we set the search space of depth to 2–12, the resulting trend analysis graph revealed a positive correlation between the depth and the validation set performance, as indicated by the Pearson correlation coefficient (r). This analysis suggests that higher values of depth tend to yield better performance. Therefore, a refined strategy would be to narrow the search space to values between 8 and 11 for further tuning.
Separating the experimental spaces based on the purpose of the experiment not only makes it easier to track and find specific experiments but also, in the case of hyperparameter tuning, helps in uncovering valuable insights that can significantly improve model performance.
Assessment on Organizing Experiment Management
In our AI project, organizing experiment management became a crucial task. Here's what we learned about structuring our experiment management and how it has helped us advance the project.
What Worked Well: Seamless Collaboration and Efficient Comparison of Experiment Methodologies
With an organized experiment management system, collaboration across the team became more seamless. For instance, a teammate might say, “I tried an experiment with Method A, and here are the results,” then share a link to the MLflow experiment. Clicking the link reveals a detailed description of Method A, the data version used, and the source code version. This setup allows me to read the description, modify the code in that specific source code version, and run a similar experiment with slight variations. Without this system, re-implementing Method A would be inefficient, potentially leading to inconsistencies and difficulties in comparing the performance of newly developed models.
Beyond facilitating collaboration, this approach also helps us identify directions to improve model performance. By comparing the average performance between methodologies, we can focus on comparable experiments rather than sifting through a disorganized collection of experiments.
What To Improve: Inefficiencies and Human Error
While our experiment management system improved organization, we also focused on maintaining source code versions for all experiments. The model development cycle typically followed this pattern: conceive a methodology → implement the methodology in code → submit a pull request → review the pull request → run the experiment. However, this cycle sometimes introduced inefficiencies, especially when quickly trying and discarding various methodologies. To address this, setting clear criteria for quick trials and establishing exception cases can help avoid unnecessary processes and streamline experimentation.
Additionally, we created a src_version key in the config file to log the source code version in MLflow and recorded it as an MLflow Tag. However, manual modifications to the config file during experiments led to human errors. Automating this process as much as possible is essential to minimize errors and enhance efficiency.
In this blog, we’ve discussed the importance of experiment management and how creating an effective experiment management system helped us successfully complete our project. While the core principles of AI projects remain consistent, each project may have slightly different objectives, so it’s beneficial to customize your experiment management system to suit your specific needs. In the next post, we’ll share more about how MakinaRocks’ experiment management expertise is applied to our AI platform, Runway, to eliminate repetitive tasks and increase the efficiency of experiments.
We stand on the brink of a new era, fueled by the rapid advancement and integration of Artificial Intelligence (AI). Today, the manufacturing industry is poised to undergo a transformation unlike any it has seen before.
While the transition from manual labor to automated processes marked a significant leap, and the digital revolution of enterprise resource management systems brought about considerable efficiencies, the advent of AI promises to redefine the landscape of manufacturing with even greater impact.
Central to this transformation are Large Language Models (LLMs) and generative AI technologies. These tools are significantly lowering the barrier to entry for subject matter experts and field engineers who traditionally have not been involved in coding or "speaking AI." The impact of this should not be underestimated. Up to 40% of working hours across industries could be influenced by the adoption of LLMs, a significant shift in workforce dynamics.
AI, and particularly LLMs, will have a profound impact on the manufacturing sector. The opportunities are vast — but there are potential challenges, too.
AI is reshaping the very fabric of manufacturing, transforming traditional automation frameworks that adhere to ISA-95 standards at every level. This new era of automation heralds increased productivity and the emergence of innovative manufacturing practices, all driven by AI.
The integration of hardware automation, spearheaded by advances in robotics, combined with software automation led by AI, is crucial to unleashing the full potential of these innovations.
Yet, despite these advancements, AI remains an alien concept to many within the manufacturing industry. Subject matter experts, the seasoned engineers who intuitively understand machinery and production processes, find themselves at a crossroads. As these experts retire, their invaluable knowledge and insights risk being lost, underscoring the need for AI's integration into manufacturing to bridge this gap.
LLMs are set to revolutionize the manufacturing industry by serving as a conversational gateway between humans and machines, enabling assets and machinery to "communicate" with humans.
By interpreting vast amounts of manufacturing data, LLMs facilitate informed decision-making and pave the way for the future use of natural language in production and management.
This symbiotic relationship between AI and humans enhances the intelligence and efficiency of both parties, promising a future where AI's impact on manufacturing is more transformative than the industrial revolutions of the past.
In this future, AI amplifies human expertise, creating a collaborative environment where decision-making is faster, more accurate and informed by insights drawn from data that was previously inaccessible or incomprehensible.
An industrial LLM encapsulating all layers of the manufacturing plant, from machinery to AI-driven analytical solutions, will be able to manage and optimize entire operations.Image: MakinaRocks
The integration of AI into manufacturing extends beyond simple automation, encompassing areas like control optimization. By analyzing vast datasets, AI enhances production efficiency and reduces costs through the optimization of manufacturing processes. This not only smooths operations but also minimizes resource waste.
Reflecting the importance of these technological advancements, research shows that 75% of advanced manufacturing companies prioritize adopting AI in their engineering and R&D strategies. This commitment underscores AI’s key role in the future of manufacturing, guiding the sector toward more efficient and sustainable practices.
In the not-too-distant future, AI will be able to manage and optimize the entire plant or shopfloor. By analyzing and interpreting insights at all digital levels—from raw data, data from enterprise and control systems, and results of AI models utilizing such data—an LLM agent will be able to govern and control the entire manufacturing process.
For AI and LLMs to truly transform manufacturing, they must first be tailored to specific domains. This customization requires not only connecting to the right data sources but also developing tools for effective prompting that align with the unique challenges and processes of each manufacturing sector.
Domain specificity ensures that AI solutions are relevant, practical and capable of addressing the nuanced demands of different manufacturing environments. This demonstrates the need for industrial LLMs (or domain-specific LLMs) for proper and accurate application of LLMs in manufacturing.
In addition to domain-specific tailoring, the widespread and successful adoption of AI in manufacturing necessitates standardized development and operational processes. Establishing common frameworks and protocols for the implementation of AI technologies is critical to ensure compatibility, interoperability and security across different systems and platforms.
Standardization also facilitates easier adoption and integration of AI technologies, helping manufacturers to navigate the transition to AI-powered operations with greater ease and efficiency.
The AI transformation in manufacturing is set to usher in an unprecedented level of innovation. To keep pace with this rapid advancement, manufacturing leaders must make timely and informed decisions.
Preparing for this shift means implementing organization-wide AI transformation initiatives to standardize the AI development and operations processes and laying the foundation to fully leverage the benefits AI offers.
As the manufacturing industry stands at the cusp of this new era, the integration of AI promises to bridge the gap left by retiring experts and propel the sector towards a future of unparalleled efficiency and innovation.
The journey towards AI-enabled manufacturing is complex and fraught with challenges, but the potential rewards make it an endeavor worth pursuing.
Explore further insights on the significance of large language models by visiting the World Economic Forum (WEF) website.
At MakinaRocks, we specialize in anomaly detection, amongst other things. But what exactly is anomaly detection? In this post, we will explore a few standard methods used to conduct anomaly detection and introduce our approach.
Anomaly Detection, also known as fraud or novelty detection, is the classification of normal and anomalous data. Anomaly Detection is essential in situations such as credit card fraud detection, video surveillance, autonomous driving, and industrial machinery maintenance.
Binary Classification
As the name suggests, binary classification models predict the classification of two different classes. For instance, binary classification can be deployed to determine if something is spam or not spam, correct or incorrect, or any two binary classes. To this end, it is widely used to perform anomaly detection.
The image below depicts classification, in which there is a boundary separating normal and abnormal data, and anomaly detection, in which anomalous data refers to data distributed outside of the normal data range.
However, in the highly dynamic conditions of the real world, anomalies are not merely binary. Anomalous activity typically occurs sporadically, in abnormal patterns. Even if we were to define an anomalous class by training the model with anomalous patterns, we would still not be able to narrow down and learn all of the patterns of the anomalous data.
Principal Component Analysis (PCA)
Another common method of anomaly detection is Principal Component Analysis or PCA. A “special” form of autoencoder, PCA maps material from high-dimensional spaces to low-dimensional spaces with Singular Value Decomposition (SVD), as depicted below.
During PCA, features can be extracted and compared to determine if anomalous or not through linear dimensionality reduction. However, due to the limitations of dimensionality reduction, PCA is not always the most viable solution. In cases such as the one depicted below, anomalous samples cannot be detected.
Semi-supervised learning
Semi-supervised learning algorithms are trained from a mixed batch of labeled and unlabeled instances.
An effective method of improving accuracy, semi-supervised learning, is widely implemented to detect and classify anomalies. They can be classified into two different cases: unimodal and multimodal normality cases.
Unimodal normality cases, or one-class classification, refer to situations in which normality is represented by a single set of normal features. This is exemplified by the image below, in which we have a set of “normal” MNIST samples on the left and anomalous samples on the right.
With countless factors to consider, creating a model for the real world is a challenge, and normality in the real world cannot be defined by a single set of patterns. Given the dynamic conditions, real world problems are generally defined as multimodal normality cases.
We will take a car engine, for instance. If we were to perform anomaly detection on a car engine, the engine could be defined into four different states: intake, compression, explosion, and exhaust. As each state differs from one another, a more complex method is required to train and evaluate a model for this problem.
We can better explain the model training and evaluation process for said scenarios with MNIST. For instance, model training can be performed with nine classes. Once the training is complete, the remaining “1” class, or the anomaly, can be used to evaluate if the model accurately classified the cases.
Due to the complex nature of this method, training is difficult to conduct, and anomaly detection models have been known to show lower performance.
Autoencoders (AE)
An autoencoder extracts features with dimensionality reduction--just not linearly. What an autoencoder does is to extract features by compressing and decompressing data. Let’s take an MP3 recording, for instance. When you listen to an MP3 recording, you’re actually listening to data compressed after discarding data in the frequency range not perceptible to the human ear.
In doing so, autoencoders teach themselves how to extract features through the process of encoding and decoding. However, due to the MSE loss function in which the autoencoder predicts the average value for uncertain areas, results are often unclear.
Generative Adversarial Network (GAN)
There are two components of the Generative Adversarial Network: the generative network and the discriminative network. The generative network learns to create “fake” examples while the discriminative network learns to distinguish said “fake” examples from real world examples. Since the model is trained through the hostile learning between the generator and the discriminator, it does not possess a module that performs dimensionality reduction.
To conduct anomaly testing, a module for dimensionality reduction is necessary. While existing GAN-based anomaly detection methods suggest various methods to solve this issue, the balanced learning of the generator and the discriminator and the shortcomings of MSE act as a significant obstacle.
References [1] Ki Hyun Kim, Operational AI: Building a Lifelong Learning Anomaly Detection System, DEVIEW, 2019 [2] Jinwon An et al., Variational Autoencoder based Anomaly Detection using Reconstruction Probability, SNU Data Mining Center, 2015 [3] Anh Nguyen et al., Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images, CVPR, 2015 [4] Ian J. Goodfellow et al., Explaining and Harnessing Adversarial Examples, Arxiv, 2014 [5] Ki Hyun Kim et al., RaPP: Novelty Detection with Reconstruction along Projection Pathway, ICLR, 2020 [6] Stanislav Pidhorskyi et al., Generative Probabilistic Novelty Detection with Adversarial Autoencoders, NeurIPS, 2018 [7] Lukas Ruff et al., Deep One-Class Classification, ICML, 2018 [8] Siqi Wang et al., Effective End-to-end Unsupervised Outlier Detection via Inlier Priority of Discriminative Network, NeurIPS, 2019 [9] Thomas Schlegl et al., Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery, Arxiv, 2017 [10] Houssam Zenati et al., Efficient GAN-Based Anomaly Detection, Arxiv, 2018 [11] Ilyass Haloui et al., Anomaly detection with Wasserstein GAN, 2018 [12] Izhak Golan et al., Deep Anomaly Detection Using Geometric Transformations, NeurIPS, 2018
You can’t talk about the industrial sector without talking about industrial machinery. Neither can we. At MakinaRocks, we aim to make industrial technology intelligent and deliver it as transformative solutions.
We believe in tailoring our AI solutions to meet the needs of our clients, which not only includes increased technical specifications involving efficiency and performance metrics, but something just as important: convenience.
There are a number of ways to maintain industrial machinery. If you search around, you will most likely come across these terms: preventive maintenance and predictive maintenance.
While both refer to preventing industrial machinery breakdown (and factory shutdowns – yikes!), preventive maintenance generally involves scheduling an expert to come down to your factory once every few months to take a look at your machinery and see if anything needs fixing. On the other hand, predictive maintenance refers to using data to predict when the machine will break down, so you can call in an expert prior to the anticipated breakdown. MakinaRocks solutions take the concept of predictive maintenance a few steps further.
What we’ve built is a predictive maintenance solution based on AI—excelling in accuracy and performance.
In this post, we will explain our novel anomaly detection metric behind our state-of-the-art anomaly detection solution: Reconstruction along Projection Pathway (RaPP), a concept acknowledged by the International Conference on Learning Representations (ICLR) in 2020.
What is RaPP?
RaPP is our proposed anomaly detection metric for autoencoders—drastically improving the anomaly detection performance without changing anything about the training process.
RaPP redefines the anomaly detection metric by enhancing the reconstruction process. Reconstruction is the difference between the input and output of an autoencoder. RaPP extends this concept to what we refer to as the hidden layers in our paper. This is done by feeding the initial output to the (very same) autoencoder again and aggregating the intermediate activation vectors from the hidden layers.
SAP & NAP
To reiterate, RaPP enables the comparison of the output values produced in the encoder and decoder’s hidden spaces. We implemented two different methods to measure performance: RaPP’s Simple Aggregation along Pathway (SAP), and RaPP’s Normalized Aggregation along Pathway (NAP).
Figure 2 exemplifies SAP, but when the distribution is so, it merely depicts the distance from the origin. A more suitable approach would be to calculate the distribution as exemplified in figure 3, which is equivalent to achieving the Mahalnobis distance. To do this, we can attain a normalized distance by applying Singular Value Decomposition (SVD) to the hidden reconstruction error of each layer from the training set. This concept is also known as RAPP’s NAP.
With SAP and NAP, a more accurate anomaly detection process may be performed by producing an anomaly score with the scalar value from the difference between multiple layers. Please refer to our paper for a more in-depth look at our methodology.
Our RaPP results
We performed numerous experiments to verify the effectiveness of RaPP. We began by using widely known and accepted datasets such as MNIST and FMNIST to compare the performance of our model with research from published, peer edited papers. The results are as follows:
As you can see from the results, when NAP is applied to different autoencoders, we found that our results were far superior to the results in published papers. Namely, when NAP was applied to Variational Autoencoder (VAE), we were able to see the most significant results.
Further, we were able to prove the effectiveness of RaPP in multimodal normality cases as shown in the results below.
Upon implementation of RaPP, all but STL (steel) showed improvement in performance in multimodal normality cases. RaPP also proved to be more effective in 6 out of 10 instances in unimodal normality cases.
MakinaRocks’s transformative solution for predictive maintenance: ADS
Our RaPP anomaly score metric is featured in our Anomaly Detection Suite (ADS), empowering the solution to predict machinery breakdown with increased accuracy.
For more information about ADS, contact us at contact@makinarocks.ai
Want to know what we do? Visit us at www.makinarocks.ai
To read our ICLR 2020 accredited RaPP paper: https://iclr.cc/virtual_2020/poster_HkgeGeBYDB.html
References
[1] Ki Hyun Kim et al., RaPP: Novelty Detection with Reconstruction along Projection Pathway, ICLR, 2020
[2] Lei et al., Geometric Understanding of Deep Learning, Arxiv, 2018
[3] Stanislav Pidhorskyi et al., Generative Probabilistic Novelty Detection with Adversarial Autoencoders, NeurIPS, 2018
[4] Kingma et al., Auto-Encoding Variational Bayes, ICLR, 2014
[5] Makhzani et al., Adversarial autoencoders. Arxiv, 2015.
[6] Raghavendra Chalapathy et al., Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.
[7] Lukas Ruff et al., Deep one-class classification. In ICML, 2018.
[8] Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. NIPS, 2018.
[9] Ki Hyun Kim, Operational AI: Building a Lifelong Learning Anomaly Detection System, DEVIEW, 2019
