using machine learning and deep learning for energy forecasting with matlab -凯发k8网页登录

good morning. welcome to the webinar on using machine learning and deep learning for energy forecasting with matlab. i am jayanth balaji, i am an application engineer here are mathworks. let us quickly deep dive into the topics that we have for today's discussion. in today's talk, we will learn what is load forecasting, who is benefiting by doing it, and why we need it. how to leverage your domain expertise in the ai workflow using matlab, and finally we will learn how to deploy the algorithms seamlessly to enterprise scale solution and integrate them with a dashboard.

let us quickly jump into the first bucket for today's discussion on what is load forecasting and who will be benefiting by doing it? why we need it? the [inaudible] is focused on the modernization, which is a megatrend seen in the recent years. when you think about the grid modernization the first and foremost thing that comes into play is renewable integration, which can be pv, wind turbine, and things like that. due to the integration of renewables, there is a strong end requirement for energy storage because of both renewables and energy storage in the grid creates a lot of uncertainty in the grid operation.

through the process of the entire grid, starting from generation, transmission, and all the way down to distribution, we generate terabytes of data in the deep cost of operation. and finally, the regulatory oversight comes into a picture of the entire power grid. the entire grid modernization can be conceptualized as almost smart grid, which primarily focuses on three area. one is power system studies and compliance study, grid analytics, and finally iot and ot applications. as part of today's talk, we will be focusing more on grid analytics.

grid analytics is a huge focus area which starts from asset management that will monitor the transformers and underground cables and things of same sort. we can develop the predictive maintenance algorithm, and we can estimate the remaining useful life of those assets. similarly, you can think about accessing the grid stability and integrating the osi pi with matlab, and doing some online analytics on top of the data, which we acquire from the field. we can also think about energy and price forecasting, but we can develop a [inaudible] algorithm to do the load forecasting and thinks of same sort. as part of today's talk, we will be focusing more on energy and price forecasting bucket.

before getting into the actual topic for today's discussion, let us look into a user story from fenosa where they predict energy of supply, demand. using mathworks tools, they develop a flexible model, which can able to be updated and optimized based on their needs. and here are the results which fenosa achieved. they reduced the response time by months, their productivity has doubled, and program maintenance is highly simplified.

let us look into what is energy forecasting. energy forecasting can be type of load forecasting, predicting load on the grid, or enable forecasting such as wind forecasting and solar forecasting, or it can be thought of price forecasting. who need this particular energy forecasting is that starting from generation companies, [inaudible] power, all the way down to electricity consumers. as part of today's topic, we will be focusing more on load forecasting and how to do it in matlab, and what are the challenges, typically, comes when you think about load forecasting and how matlab comes in handy in those multiple case scenarios.

when we think about energy forecasting, even multiple utilities today, they are doing it manually. the energy forecasting workflow, which is done manually, is not scalable and not label in most of the cases. we can think ai as a tool to our trusty energy forecasting workflow and to make it more reliable and scalable. there are four buckets involved in ai-based workflow, starting from accessing, exploring the historical data, pre-processing the data, developing the forecast model with the help of machine learning and deep learning, and finally operationalize the models by deploying them into cloud or embedded systems.

there are multiple challenges associated with energy forecasting workflow starting from aggregating the data from multiple sources and handling the big data, cleaning the data and pre-processing it so that it is suitable for developing the forecast model, choosing a right forecast model from the number of forecast model that is available, and how to scale them into the production scale. those are the typical challenges associated with energy forecasting workflow. let us deep dive into each and every bucket on how matlab solves the challenges easily and helps you to do the task at a higher productivity.

in today's case study, we are going to develop machine learning and deep learning-based load forecasting algorithm to do the day ahead load forecasting. we are going to consider the new york iso data set, which comprises of [inaudible] and feeder in it. we will develop the load forecasting algorithm in matlab. we will debug them, and make it suitable for deployment. we will deploy them into matlab production server, which is cost on azure. we will integrate it with the third party dashboard and get the insight [inaudible].

when you think about the challenges associated, we have multiple challenges starting from data acquisition and pre-processing, finding the right set of forecasting technique, and scaling them to the cloud deployment. in big picture, we can-- finally, we are going to see how to integrate the developed algorithm into dashboard and get the insights, good visualization, of next day.

let us look into each and every bucket of energy forecasting workflow starting from accessing data all the way down to operationalizing the models. let us focus on the first bucket, accessing and exploring the data. these are the typical challenges associated with this particular bucket starting from getting the data from multiple sources, visualizing them. let us look into matlab-based workflow for this particular bucket.

before getting into this, this is a typical matlab window which comprises of current folder, workspace for the current working variables, command window where we will do the instant commands and testing, and this is a notebook which comprises of a text, which comprises of codes. we can add live task. task is nothing but a gi-based workflow. we all know how life controls where we dynamically tune the particular input and get the results [inaudible]. let us look into the first bucket, accessing and exploring the data.

i'm getting the historical data, as i mentioned, from new york iso. i am downloading the historical data from website. i am unzipping and saving it for my further processing. here i am [inaudible] using a data store, which is a pointer to your data, which handles the big data effectively on [inaudible] data challenges such as memory workflow and things like that. i am getting the raw data here. let us look into the raw data. you can see that multiple figures are there, which is not properly organized. and there are a lot of zeroes in that.

let us unstack them to get a better idea how one each and every figure looks like with the help of matlab command code unstack, prepare the command. i'm just unstacking the results. now, i'm getting the better view of how the data looking for each and every figure. we also have a dedicated task based workflow for this unstack if we are not-- i am just using the one of the inbuilt tasks that is available in forms of tables and timetables bucket, which is nothing but unstacking of tables. i am just opening that.

once you open the live task, it pop [inaudible] a gi window in the matlab. that we are going to select the table, which we want to unstack them. in our case, we are going to open the raw data set, which is nothing but nyiso raw. and i am selecting which will be the new name column and value column. once i select that, matlab, in background, runs the program for me and gives me the similar result, which we didn't program that way. [inaudible] stop there, you will get the column code, whatever we do in form of task. so that there is nothing black box, we can understand what is happening behind the scenes and plan our things accordingly.

to make sure that is one and the same, i am just opening the table, which is table i've been using with task, which is nothing but the same table as we generated using a program [inaudible]. let us quickly [inaudible] our signals to understand how well our data looks like, which gives an insight, also, that how our data points are there. i am going to use the stacked plot command, which is available in part of matlab predefined plotting command.

so i can see that plot off all the grids time. and just by visualizing, i can understand that there is some abnormalities in the data. here, you can see that there are negative log conception, and there are multiple zero points in the data. by plotting what we understand here is that the data is not so clean and it requires a lot of preprocessing to be carried out.

i am just checking whether my data is the regular or equally sampled just with the [inaudible] concept called isregular command which is available in matlab, which gives me answer of zero, which means that my data is not regularly sampled. so we understood that a lot of abnormalities in the data in form of zero power consumption or negative log, and also the data, is not equally sampled or regularly sampled.

so coming back to the challenges, we start with this particular workflow. starting from how to access the data from multiple sources, on how to visualize them, and we see that how matlab comes in handy in this particular bucket and solve the workflow at much of ease. so the challenges are normal challenges in this particular workflow. let us look into the other buckets and discuss in detail.

let us look into the next bucket, which is nothing but pre-processing and analyzing the data, which is one of the most difficult bucket in this entire workflow. it is a quote from forbes, which says that 80% of the entire forecasting workflow, or the data analytics workflow, is spent on this particular bucket. and typical challenges associated with the workflow, starting from data aggregation from different sources, cleaning the data which is poorly formatted, irregularly sampled, and with redunadant and outliers. this typical bucket also has a problem with dealing out of memory problem, which we dealt earlier bucket as well.

let us look into how matlab works in this particular bucket. when we think about pre-processing and analyzing the data, which can be sub-bucket into four workflows or four segments, starting from cleaning missing data, cleaning the outliers, smoothing the data, regularizing the timestamps. earlier in our data visualizing, we have identified that our data suffers from a lot of zero entries, which is not a proper one. here, we are going to use matlab live task, which is nothing but a gi-based workflow, to do the pre-processing bucket.

when you see here, we have multiple live task, especially in the data pre-processing bucket. we have clean missing, clean outlier, and smoothing of the data. i'm going to select one such task, which is nothing but clean missing data. what i'm going to do post-selecting is that i'm going to select the table which i want to clean. i'm selecting the input data, which is nothing but unstacked data. i'm selecting the variable in the table, which is nothing but one of the feeder unit. i am selecting a [inaudible] as a feeder unit here. i'm selecting the cleaning methods. and multiple interpolation techniques that are available to do this particular task, here you can see that.

i'm just selecting the linear interpolation. once i save that linear interpolation, it automatically does the aurofilling of the missing entries. an advantage in autolearn stops here, we can able to see the innerlying code associated with this particular workflow. so there is nothing black box. we can deep dive into each of this code and understand better. and the next is nothing but cleaning the outliers. i'm going to use the similar workflow as we have a lot of spikes in our data. i'm going to use the one more live task, which is nothing but the clean outlier data task.

i'm going to import the previously cleaned the data, and i'm selecting the cleaning methods and suitable interpolation techniques. i'm specifying that threshold fact, which is nothing but the standard deviation. you can see, over here, dynamically the plot and the outlier is removed. you can appreciate that. however, it is easy to do with this task. since my data is also susceptible to noise and have a lot of noise in the data i am going to smoothen it. i'm importing the outlier remove data, and i am selecting the smoothing parameter and smoothing method. again, you we able to see the code, inner line code.

and as i mentioned in the previous bucket, our data is not equal sampled. we are going to use regularized timestamp. [inaudible] timing task that is available, again, in the live-- as part of live task. i'm going to import this method data. i am going to resample it for 30 minutes for my further processing. and here, i am getting the data. you can see here that it is equally sampled, now, for 30 minutes. i can see a lot of missing entries even here. i can able to fill that dynamically with the help of just clicking and pointing to the right filling methods. and again, i can able to visualize the inner line code of that.

you can appreciate how it is easy to do the pre-processing step with the help of matlab and its task. and there is nothing out of black box-- and there is nothing black box in it. you can understand the inner line code of it. the next step is i am going to aggregate the data since we know that load is depend upon the weather data also. i have done similarly cleans to really clean the weather data. i am loading it to clean the data. i am going to merge them together to create my predictors for my load forecasting workflow. i am just using the enable commands in commands to join them. we also have a dedicated task-based workflow which does the same job for you without doing any program, just by doing it interactively. and i am selecting the suitable joining mechanism over here. once that is done, i am able to synchronize the two tables.

so this is done. again, we are understanding inner line code of the [inaudible]. and i'm going to extract the predictors. so what i mean by predictors is that, fundamentally, we know that the load data depends upon multiple factors such as hour of the load conception, month of the load conception, and year of the load conception and things like that. so we are going to extract features from the data. we are going to extract features for our machine learning-based workflow. and we are also handling the time series data so that a sequence dependency exists in the workflow. so we are just plotting the correlation workflow-- correlation plot and understanding what is the typical pre-sequence to be considered for this particular time series forecasting.

so we have done that, and we understand that, typically, prior day is considered. and i'm generating the feature extractor to do my machine learning-based workflow. so you can think about it starting from a pre-processing step of cleaning the missing data all the way down to creating the flow. it's done in matlab and vary between very minimal programming knowledge. in a fraction of a minute, we can able to complete this particular workflow.

let us look back into the challenges of this particular workflow, which we discussed earlier starting from data aggregation from different sources and cleaning the data effectively, and handling the big data. and as for the forbes quote, we identify that this is one of the difficult bucket. by using matlab and its task, we have done the job very easily. you can appreciate that matlab-based workflow, how does the job effectively and easily for the entire workflow. for this particular workflow of a pre-processing and analyzing the data, the challenges are normal challenges. you easily adjust with help of matlab and its tools.

let us look into the third bucket of the energy forecasting workflow, which is nothing but developing the forecast models. so when we think about developing the forecasting model, we can think about multiple techniques to address this. so we are going to focus on machine learning and deep learning-based workflow. autofetch machine learning workflow requires feature extraction, which we did in previous step by extracting the predictors. and deep learning is nothing but a subset of machine learning, which doesn't require any feature extraction. it operates on pre-processed data.

when you think about typical challenges associated with this bucket, most of the people like us or lack data science expertise, and we don't know the right tool to solve the problem. and the required amount of time required for feature extraction, feature selection on model development of this particular workflow, is very difficult and time-consuming process. let us look into the first bucket of this particular workflow, which is nothing but a machine learning-based workflow, and let us look into a matlab-based workflow. how it solves the entire process.

so a typical question arises within our mind is that, i am not a data scientist, how to develop an ai model for my workflow? for that, we have a dedicated app-based workflow inside matlab. when i look into the apps, we have multiple apps inside the machine learning and deep learning workflow. i am just selecting one such app, which just nothing but the regression learner. since we have a task as a forecasting task, a regression-based task, i am just loading the data, which is a pre-processed data which we did in the earlier bucket. and i am selecting the response and predictor variable. here, my response variable is nothing but a load and predictor table is the features, which we extracted in the previous step.

since my data is time series data i am not selecting any validation, but you can select based on the sequences length you are considering. once the data is loaded in this particular app, matlab automatically populates the rest of suitable algorithm for your workflow. so here, you can see that exhaustive set of algorithm that you can try it out starting from linear regression all the way down to ensemble trees. i'm going to try out all algorithm that is suitable for my data. i'm just pressing all and i'm enabling my parallel tool. i'm just-- i'm hitting the train model.

this is nothing but an automl concept. but we need to fill the data. just everything will be taken care of by this particular app. there are 19 models in total in order which folder is running parallel. because i have my quad formation. in a fraction of a minute, i can able to get a reasonable algorithm with reasonable forecasting accuracy, which is nothing but svm i am getting for this particular data i got in minutes.

but i am not satisfied with my accuracy. matlab provides a capability to finetune this particular algorithm. i am selecting optimizable svm since svm is found to be suitable for my data. i am going and selecting the optimizable svm. once i select the optimizable svm, i can able to select the hyperparameters i want to tune and type of optimizer, or type of optimizer which i want to try it on the actual hyperparameters in this particular workflow. i am selecting a bayesian optimizer, since it provides faster results.

i can also file select, which are the hyperparameters i want to tune starting from kernel functions and standard data. i am just selecting run again. so this will finetune the algorithm, which we got with reasonable accuracy in the earlier stage for the better accuracy. in fraction of minute, it's also got converged with the suitable hyperparameter for my data. the advantage in autolearn stops here. we can able to generate the function and we can understand what is happening inside for this particular workflow. and we can also export the model back to matlab workspace for the deployment. so machine learning is easily done with help of matlab.

but what if you don't have an idea to extract suitable features for your workflow? that is one of the crucial step in the machine learning-based workflow. so you can think about deep learning in that case. if you don't have enough domain expertise or even enough time to extract features from the data, you can think about deep learning. when you think about deep learning workflow, we are going to try out something different from the previous workflow. in the previous workflow, what we did is we try out individual models for individual feeder in the particular iso. here, we are going to play out unified modeling approach where a single model can able to address a multiple feeder at a time, and it can able to generate a forecast at time for all the feeders under consideration.

one quick change. what here we are doing is that we are splitting the data in form of seasons. this approach is applicable even for machine learning. just to give an idea that it is applicable to all-- the one and all, i'm just trying it only for deep learning. i have split up the time initially. there our data after pre-processing. and since deep learning operates on normalized data, i just want to normalize the data. for that, i'm not doing anything. and i am generating diligent lag generation as we did in previous case. it is a prior data.

that said, we are not doing any feature extraction step in this particular workflow, deep learning-based workflow. we are going to use the same data for developing the deep learning models. so there are multiple techniques for developing deep learning, especially on time series data. so we are going to try out lstm in this particular example, our particular case study. to solve the deep learning-based workflow, so again, we are going to use the app-based workflow. in particularly, especially on machine learning and deep learning bucket. we are going to use a deep network designer by which you can interactively generate the deep learning network.

so i am just using a particular app, which is nothing but a deep network designer. once i open the app, you can able to see a lot of predefined algorithms are pre-trained network available for you. in this particular case, i am going to train the network from scratch. and i am pulling out the event layer suitable for my workflow, in this case, sequence input layer. i see ssl, i am going to try out lstm this particular example. i am going to pull out lstm layer. just to start with, i am pulling out two lstm layers. then i'm going to connect fully connected layer. then i am going to add the integration layer at the output layer.

post-selecting the suitable layers, i'm just making the connection between the layers just by tagging and clicking. this particular app allows you to interactively chain the input and output parameters. in our case, we are having 528 inputs and 528 outputs. i am just manually doing it. and this app also comes with analyzer, which analyzes the network for error. in this particular case, i don't have any error. it also gives an idea how input travels from input to output layers and different learnables associated with this particular network.

post-analyzing, i can export back the network to matlab for training with initial parameters or just the network. i just exported back the network in matlab, you can see that the entire network is built in form of like an app-based workflow. this is the same network i have copied here. i am specifying detaining options, which is nothing but hyperparameters. and i'm going to train the network. it help a command called trainnetwork construct available in matlab. i feed the input, output layers and options. and i'm just saving the network in the name of net dot spring.

this is a training graph. and i obtained 400 epochs, which gives the input and starting time, and estimate time, or elapsed time. i added similarly for all the seasons, and i obtained the network. you can appreciate how the deep network [inaudible] helped in this particular workflow of generating the deep network. i want to try out multiple experiments that is on the hyperparameters of this particular network. in this particular network, the hyperparameters are nothing but the multiple options. you can try out-- to try out multiple experiments on the network and comparing the best of the [inaudible] for our workflow. you have a dedicated workflow called experiment manager.

but i want to create a new project for comparing my experiments and select the best hyperparameter for my workflow. i am just planned to set my experiments on my initial learning grid and my epochs for training. once i set this experiment, i can go ahead and run my experiments on my developer network in the previous stage. so i'm going to just run the experiment on my network. you can see that the experiment is running on my network, and this is a snapshot of a completed experiments or completed workflow.

there, you can see that multiple experiments are on the same networks with multiple combinations of learning rate and epochs, and their autumn season losses are plotted over here. we can select the best work suitable for our workflow. and we can export a particular network back to matlab for the next stage, which is nothing but the deployment. i can filter it out, i can select deep suitable network based on my choice, and i can export back to matlab.

[inaudible] or a trade-off before going into like a machine learning or deep learning. here, how to make a choice whether you want to try out deep learning or machine learning. we exhaustively support both of these workflows. based on your domain knowledge and data sense-- data set availability, you can make a choice whether you want to go ahead with deep learning or machine learning. we also support intermittent workflow, that we extract some features to assist the deep learning in form of time frequency map or pre-processing. we also support them exhaustively inside matlab. we can try out multiple things. and this is a great-- we'll give you an idea how to choose the choice of algorithm for my workflow. so it depends upon data set size versus domain knowledge versus compute resource you have.

let us look into the final bucket of the energy forecasting workflow, which is nothing but operationalizing model. we have developed the forecast model in the previous bucket, and now we want to deploy them to the-- deploy them to the cloud or embedded hardware for operationalizing them. so let us look into a quick chart, our quick workflow chart, which gives an idea of what we can do once the model is developed in the matlab. we have multiple products aligned to support the operationalizing model. we can think about a coder-based workflow. if you want to deploy the end algorithm and embedded hardware using the coder product, you can able to generate the c, c , hdl, plc, and coder-targeted codes.

using the compiler-based workflow, you can able to generate the standalone executables like exes. you can generate excel add-ins. you can also try out matlab webapp and webapp, by which you can host the app in your browser. we also have matlab compiler sdk, by which you can able to generate the shared library package, such as c, c , java, python, and .net. this comes in handy if you have a larger system which is already developed in these packages and you want to integrate this particular energy forecasting with matlab. you can [inaudible] to generate the shared library and you can integrate with the existing system.

we also support cloud-based deployment with help of matlab production server. there we can hold-- there which can-- where matlab production server can host multiple matlab algorithms on cloud-based platforms. when you think about the challenges, there are multiple challenges associated with this particular bucket starting from how to scale the app and how to deploy them. traditionally, people do the manual translation, of course. and the end user can be highly varying in nature, starting from operators onto all the way to customers. so there are multiple targets, which will [inaudible] target platform. so we have all the way from cloud to embedded hardware.

so there are multiple challenges associated with this particular workflow. let us look into quick demonstration with help of matlab how to solve these challenges. we have a matlab function, which in nothing but the forecast model, which we did-- which we created in the earlier workflow. we have multiple workflow. then we are going to-- we use the inbuilt app, which is nothing but a matlab production server compiler which comes as part of application deployment app. i am going to select the matlab production server compiler by which we can be able to generate the cloud executable.

i'm just opening the function. this app is very intelligent enough to understand the prerequisite automatically. so in this particular app comprises of all the predefined models to be used in this particular deployment. it automatically recognized that. i'm just testing the client before posting it on the actual cloud. and i am selecting the matlab and functional matlab breakpoint errors. and i am just now starting the client. it's just nothing but the matlab app is hosted on this particular endpoint. and i'm going to write a few lines of code which is already available in matlab documentation to pass the workflow.

so i'm just placing a web write on this particular endpoint. the endpoint is nothing but local host in the port number followed by a matlab function. i'm just placing a web write, which is nothing but an api call. once a press n, can able to see that the request is placed on the local host. that status is pending. once execution is completed, the status change to complete, i will get the insight. this is a debugging platform which allows us to make our algorithm error-free. we can thoroughly test it before deploying into the end cloud platform.

i'm going to package the cloud executable. i'll generate the executable, cloud [inaudible] which is nothing but a ctf, which is compile target format. [inaudible] select this particular workflow. it comes with a readme file, which discuss about the steps and procedure we can follow for deployment. as i said earlier, i am going to deploy it on azure platform where a matlab production service is hosted to run the matlab algorithms. i'm just uploading the application, which is nothing but the application which is centered on the [inaudible]-- with their matlab production server compiler. and i am selecting the deploy. it's deployed on this particular endpoint. so i'm going to use the same endpoint to place my epa calls and get the results.

so in the same testing code i'm going to use, only thing is that i'm going to change the local host to the azure endpoint. i am verifying whether i am getting the proper insight. i'm exactly getting the same thing. now, the next step is we are going to integrate with the dashboard. here, we are going to try out power bi dashboard. we have matlab connector to support inside power bi dashboard for connecting with the matlab production server. but i want to specify the endpoints, function name, and file name. and you can able to see that i am getting the-- i am placing the query from matlab-- i'm placing the query from power bi. i am getting the insights and the follow-up table [inaudible], which comprises of all [inaudible] feeders.

all [inaudible] feeders in this particular network, [inaudible] particular [inaudible]. and i am getting the normalized power consumption value for next 24 hours. the same thing i'm going to try out with the other dashboards. i'm going to try out with the post power bi dashboard. i'm following the same thing, same procedure. i'm placing an api call to the same matlab productions our endpoint. here, i'm entirely selecting the particular feeder in the map. and the load for next 24 hours is generated. i'm going to try it one more thing, like with javascript-based dashboard. this end point remain one and the same. and i'm generating the forecast for next 24 hours. you can appreciate that how it is easily generated.

so we started with challenges, with where we want to host it on cloud-based platform and we need a manual translation. but here, everything is in scene-- but the scenery changes with help of matlab. so matlab solved these challenges easily with help of inbuilt app associated with the workflow. and you can appreciate that how it is easily doable in matlab.

in summary, matlab helps you to solve the energy forecasting problem at ease starting from data aggregation all the way to scaling up to the production level. you can appreciate that how matlab and its inbuilt apps and task-based workflow solve the entire workflow process at ease. and here are the key takeaways for this particular webinar. so when you think about matlab as a workflow and matlab as a tool for your workflow, the pre-processing is no more a time-consuming process.

we here spend a minimal amount of effort of this particular bucket. and matlab enables the domain expert to do data science. you don't need to be a data science expert to do the ai-based task in matlab. we have an [inaudible] framework and where you can use the app-based workflow. matlab automatically does the data science job for you. and matlab analytics can turn on anyway, starting from embedded hardware to cloud deployment. one and the same algorithm can turn on multiple platforms with the help of quadrant compiler products. you can think about matlab as a platform rather than a tool to solve your workflow.

to advance your skill in matlab and simulink, we have multiple courses associated-- training courses associated with matlab and simulink. for data science, we have multiple courses starting from onramp courses, which is a short two [inaudible] courses. you'll get it for free. you can able to do the machine learning-- you can able to learn matlab, machine learning, and deep learning-based workflow. we also have mooc-based courses hosted on coursera in the title of practical data science with matlab specialization.

we also offer mathworks customized training for data science. if you need customized training for your organization, please reach out at training at mathworks.com we have a consulting service by which you can accelerate your entire process where our consultants will be supporting your workflow. and you can think about mathworks as your engineering partner to do the [inaudible] task. and that's all i have for today. thank you.