the use of matlab in open science video -凯发k8网页登录

my name is shubo. and this is-- this webinar has a slight philosophical aspect to it, because it's not about an algorithm. but it's about how do you make your science more open, and what are the possibilities to do so. ok. so let's jump in. so before i dive into all of the details that you can use, i wanted to give you a very high level overview first of why matlab and open science. because you might ask yourself, ok, matlab is a proprietary language. so how is that compatible with open science?

it turns out matlab is very well suited for open scientific principles. i'm going to show you how. so matlab might be a proprietary language, but it turns out that most of matlab's functions, most of matlab code, is actually open. it's transparent. you can open it irrespective of having a matlab license in any text editor and look at what was done, how it was done, and even get a reference. most of our users, most of you, most of the four million people globally who use matlab do not actually care about what is inside the matlab code. what they care about, what they do is they use the matlab environment. this famous toolbar that you all know.

and they use it to take these pieces of code, pieces of proprietary code from matlab, which they can look in and see, and they use it. they use them. they trust them. they know they work. but they use them to build applications and analyses for their own data. and this environment, the matlab environment, makes this easy to do, because you can click and point and join things in a fairly easy manner. but all the time, it's not a black box. at every stage, you see the code that is behind this specific analysis. and definitely, all of the applications, the analyses you build, is not proprietary. it's yours to share.

and in fact, we encourage you strongly-- i'm going to move me out of the way here. we encourage you strongly to share your code. so if you go onto the matlab toolbar and you click on the community button there, this will take you to matlab central, which is the mathworks open exchange for the matlab and simulink community. and this includes a free open-source tool exchange portal. you can test your coding capabilities there. you can answer challenges. you can join discussion forums with other matlab users. so it's really all about sharing and a lot of people on there.

and finally, we understand that as people move from local desktops to science gateways to supercomputing centers, matlab needs to be present everywhere. so it's about coverage. and as we were talking briefly before the webinar started, campus-wide and institution-wide licenses try to do that, get entire campuses under one license umbrella. so everybody on these campuses and across these campuses can easily share and collaborate. if you are not a part of campus-wide license-- we have 1,800 campus wide licenses at the moment. if you're not part of it, don't worry. i'll show you what you can do with your licenses. but it's about coverage, geographical coverage.

then, matlab has strong integration capabilities now with several different languages, including python, which means there is coverage across methods. if somebody you know is using a tool box in another language that you need to integrate, you can do that very easily from matlab. and finally, there is constant support. our developers build constant support for various kinds of data formats in matlab. and this is quite important, because we know that in every single domain we have specific data file formats. and the more of these data file formats we support, the more domain coverage we have.

and you can get all of these different kinds of data in your favorite matlab environment. so that's a very high level overview of why matlab is compatible with open science. i'm now going to dive into specific

offering, specific features, that you can use to practice open science. so first, i'm going to tell you how you can use your data to tell a story. and i'm going to talk about developing software effectively, source control, talk a little bit about tips on sharing tools and software easily. i'm going to talk about downloading and reusing software for free from matlab central, file exchange.

i'm going to briefly-- in detail, talk about connecting with those users who are not matlab users, how you can share code with them, collaborators. i'm going to talk about building a portable software capsules with matlab, if you guys have heard of docker, docker containers. then, i'm going to briefly touch upon big compute, big data, parallel computation, and talk about accessing matlab over the browser. and finally, i'm going to discuss having compute and data in the same place. this is actually my favorite part of the webinar, because it is directly related to what i do. i'm going to talk about science gateways. ok.

and i try to classify these into three bins, transparent, available, and accessible. although i admit, it's not a fair classification. ok. so let's start off with how you can use your data to tell a story. now here, i show you to pieces of matlab code. they do exactly the same thing. on the left, you have your traditional matlab code, well commented, but still static code. and on the right, you have what we call a live script. and which one do you think is more transparent and more easy to understand when somebody is trying to grasp your workflow? the one on the right, the live script. so a live script is nothing else than a digital notebook.

it enables you to take matlab code and other related outputs together, and this related output can be text explaining what you did, graphics, equations, and combine them into one document. you've got user control, so that the user-- ui controls, so that the user can interact with your code by changing some parameters. and you can export this into a bunch of different formats, so you can share easily. and i find this really useful when you want to communicate your research. so i did this a little bit in lab meetings, where people keep asking, what exactly did you do? how do you normalize? collaborators can be great too. it allows others to actually interact with your code.

and sometimes when you need to upload your code, documentation, and results in one document, that can come in handy. so now i'm going to show you how to come up with a live script. it's a short video. so just to introduce my matlab workspace, this is my editor, live editor in this case. and this is my workspace, just so you're familiar with how i have my ide organized. ok. so to start a new live script, go to the home tab and you can just click on new live script. there's more than way one to do this. and once you do that, you'll have a live script.

just a second, i'll get rid of these. live scripts are not .m files but .mlx files. and then you can go to the live editor, which is the toolbar with all the tools to make your live script. so first of all, you can go to text, and you can choose a heading. you can give your live script a heading, so create a simple script. and here in this case, we are going to make a very simple sine wave, nothing fancy. and you have some text explaining.

it'll take amplitude and frequency and construct a sine wave. and then, you go and click on code, and that inserts a little code block. and there you can type in your code, which is basically an amplitude and frequency variable, a time axis, and plot the sine wave. go ahead and click on run. it gives you the output, and you can have this output on the side. you can also have an inline, like i showed in the previous example. now comes the interactive part of it.

imagine someone wants to change around the frequency of your sine wave. so they can obviously go change this value and rerun the code. but a much easier way, a much more useful way of doing it, is

going to control and replacing it with a ui controller, in this case a numeric slider. so now, you set the limits, 0 and 20, and by moving the slider along, the user can choose any frequency between 0 and 20 hertz, and the output automatically updates. so it becomes quite interactive. now we're going to do something else.

we are going to do some standard data pre-processing tasks. we're going to find maxima, local maxima. you can use any other thing, but this is just a standard one. to do these kind of tasks, go to the task button. and you'll have a bunch of pre-programmed plug and play task modules, and we select find local extrema from this palette and we just insert here. and what that does is give you a gui. and you can just choose your signal, your input signal, from one of your workspace variables, which would be y. and that immediately takes it and outputs the result of this maxima detection.

and this comes back to my point about transparency. at every point, you can go to those three dots, and you can look at the code behind this pre-programmed gui. so you can learn how this was done and you can do it yourself. so there you have it. that's how you put together a live script. you can put in equations, links, and you can even put in a table of contents. and it's like a digital notebook, and you can go ahead and share this with your collaborators. now, if you're interested in more live scripts, we've got a live script gallery from mathworks. i've shown you the link on the slide. it's got a bunch of live scripts from different disciplines.

so if you're interested, go ahead and take a look. and a neat thing about the live script galleries is you can actually open these live script online on your software browser-- you don't need a license for this-- and run them, and download the code from file exchange. ok. next, let's talk about developing software effectively. you can use source control. and what source control does-- you've got a working copy on your desktop, on your local machine, and you've got another copy, the repository, either on a server or on the cloud somewhere. and at every stage, your local copy, when you say so, is synchronized with this repository with the result that any collaborator can just download the most updated version, the synchronized version of your code, from this remote repository.

so it keeps track of changes. and it helps you collaborate efficiently. and two of the most common source control functionalities are from git and subversion, git from github. so now you can integrate git and svn with your matlab code. you can access them from the ide, as well as from the command line using this command exclamation point and then git. ok. how do you do that? so first, from the ide, if you go to your folder view-- so you go to home, layout, show current folder, and you right click. you see souce control in the menu.

and then you go and click on manage files. you do this before you start a new project. and then you get this pop up, and you can put in your repository path. that can be a github address. it can be a local inside lab. it can be a location on your local server. and what will happen from that point onwards is that this folder will be under source control. and whenever you do something, you can go into the-- you can do it from this menu, but you can also go into the command line and just type, git add, which adds all your changes. git commit, which commits them with a specific comment from you. excuse me.

and then git push and what that does is take all of your changes and commit them, and send them to the remote repository. so now, let's talk a little bit more about sharing tools and software, using projects and tool boxes, sharing software as a project or tool box. so what a project does is it packages all the dependencies with your code inside that project. so you can try this out. you can try out an example,

matlab.project.e xample.timestable. if you type that into a command window, then you should get the matlab project toolbar to pop up.

and in that project toolbar, you see a dependency analyzer. and if you click that dependency analyzer, you see such a graph. and what this graph shows you is which pieces of code in that package are dependent on which other pieces of code. and on the side, you even get a list of all of the products, all of the matlab toolboxes, that you need to run this package.

so you can check if you have everything. if you want to share a project, then a good way to share it is by using a toolbox. what a toolbox does is it takes everything, and it spits out an executable, so that the user can just install your entire package using that executable, pretty much like you do with any program.

and to do that, it's pretty simple. you go on to the add-ons button, and you click on package toolbox, and then that'll take you a very easy set of click steps that will spit out the toolbox at the end. so now, you've developed a code effectively using git source control with your collaborators. you've made it into a project. where do you share it?

now, there are a number of platforms to share. one platform that maybe you don't know about is matlab file exchange. file exchange is one part of matlab central. file exchange is a platform for repositories, code repositories. you can get, share, and reuse tools for free.

there are about 40,000 repositories available at the moment. these are authored by experts in the field with many thousands of downloads. this is what file exchange looks like. so each of these are toolboxes that users have uploaded on the site. you have a nice search bar by discipline and subdiscipline.

so you can hone in on maybe the tools in your specific area of research. and you can also filter tools by type, so simulink models versus toolboxes versus apps. and many of you probably share your toolboxes on github. and if you do that, you can actually link them to file exchange now without having to double the work.

you can simply link github repository on file exchange. users can download using this link from your github repository, and it just gives you more visibility for your toolbox. so i strongly encourage the toolbox authors out there to go and try out file exchange. ok. now let's talk about connecting with non-matlab users. and this is really about sharing matlab. and let's face it, not everybody has a license.

matlab is not the tool of choice for everyone. so what you do when you're working in a project when you have collaborators that don't use matlab. so the first scenario is you develop software, or you are the center of the project, and you have a bunch of other people who depend on your code, or who want to use your code, and they don't have a matlab license.

there are a number of possibilities, and it depends a bit on the time frame of this sharing. so the first is short term, 30 days. if somebody somewhere doesn't have a matlab license, and simply needs to run your code on their data once, then i think the easiest way to do that would be to ask them to simply download a free matlab trial, 30 days, and run the code. ok.

however, this is of limited use. so the most common scenario is an extended project, where several universities together collaborate on an academic project. and what you can do now for academic projects is you can share your license. and this is open access basically for educational institutions.

and this is called third party access. what happens is you, as the owner of that license from a university that has that license or having an individual license, you can open up. you can give these you can give your collaborators credentials and then they can access your license and use matlab. the only thing you have to be careful about is that matlab use is on your hardware. so you have to give them some

sort of access to your hardware. or they could be visiting professors at your side. but that enables a lot of flexibility when working on these collaborative projects.

and the third option is long term. say you've developed an app or you've developed an analysis, and you want this to be accessible for everyone for a long time. one of the things you could try out there are matlab web apps. it's very simple. you go to app designer. and then you just simply convert your matlab app into a web app by clicking on share and clicking on web app.

and what that does is it converts your app into an app that can be hosted on a web server and everybody, irrespective of whether they have a license, can dial in the web address and interact with your server. now let's talk about connecting with non-matlab users in a slightly different context, which is users with other software.

so the earlier icon i showed you about you being in the center and others not having licenses is one scenario. a more common scenario when you develop software in a project is this icon here, where everybody is working together. and somebody has matlab, somebody uses python, somebody with java, and somebody is a c . so the good news here is that matlab has integrated-- has integration with a bunch of other languages.

you can call other language libraries from matlab, including python. you can also do it the other way around. so you can call matlab from another language. and again, there's a bunch of different languages supported.

i should mention here that the restful api, for web services and reading data off the web, is quite useful if you're accessing an online database. to share matlab on python, there's actually two ways you can do this. you can directly interface with matlab, something i'm going to show you. you can also take your matlab code and pack it as a python package and make it available within python.

so this is a demo to show you how to access matlab from within python. so here, i have my matlab workspace with one variable y, my matlab command window. and this is windows powershell, but you can use any other command line interface or python ide.

so let's get into python. and then the first thing you should do is to type in import matlab.engine, which imports the matlab engine. so you can use matlab functionality from within python. then let's call matlab.engine.start_matlab. and let's use eng, a variable, as an instance of this class. so now you have eng, and you can call matlab functions from eng.

so let's call a matlab function. let's call randn. and let's give it normal input like you would do in matlab and if you do that, then you see that the data type is a matlab data type, double. now let's quit this command. i'm going to show you how to connect to an ongoing matlab sessions.

to do that, in your command window, type in matlab.engine.shareengine. and what you're doing is you're sharing your current matlab session-- and you give it a name-- with python. and now from python, you can access the session, including its workspace. so now instead of saying, start_matlab, let's go and say, connect_matlab.

and let's give it the name of the session that we want to connect to, which was my session. and when we do that, we will be connected to this particular session. so now we can access variables in the workspace. and we do that using eng again. and we use the method workspace to access the workspace.

and within square brackets, you can type in the name of the variable you want to access, y in this case. and let's put that in a variable in python. and now we can go ahead and play with it. you can even plot in this way, using matlab plotting features from python.

so you can call figure using this method, just like you normally would. so you give it a name. let's call this calledfrompython, so we know this was this figure was actually called from python. and if you do that, you get a matlab window. and you can start plotting in this window.

so let's plot our variable y in this window, using standard matlab plotting commands, so color line with linestyle. and there you go you've actually done some matlab plotting from python. so this is just the tip of the iceberg. there's a bunch of other functionality.

this is a cheat sheet putting together some of the important things you should know when you do this kind of matlab python integration. if you found it really useful, check it out. check out the product pages on that. there's a lot more information as well as videos. ok. let's go now a little bit more into working with shared data and reuse-- reproducibility of research.

so first let's talk about creating portable software capsules, matlab in a docker container. so the fundamental problem is no matter how when you write your code-- sorry. no matter how well you write your code, you often get this situation that at some point, many years later, when you want to reuse your code, you figure out your code doesn't work.

and this is a big issue in research, the reproducibility crisis and all of this. and then you could have other issues. you old version might not be supported, maybe your operating system changed, the dll is missing, or dependencies are missing. and what you can do in that case is you can use a docker container.

for those of you who don't know this, a docker container is essentially a capsule or a box, in which your code, with the data it needs to run, with all of the dependencies, even some aspects of the operating system are packed inside this box. and you can just download this box, this container, from docker hub, and you can fire it up, and it'll run. it'll take some resources from your host operating system, and it will run. and the nice thing about that is now you have a portable container that runs anywhere that you go. and it's portable, it's reproducible, and reusable. and now it comes with matlab in it.

so if you want to have matlab in a docker container, the link below gives you the link to the github repository with the architecture to construct your own docker file. and if you're interested in seeing how matlab and docker runs online, i'm going to share an example with you from code ocean. code ocean is a science gateway. they encourage you, once you have a publication, to upload the code that is associated with that publication, with the idea that they're going to pack it in a container for you, and keep it on their website, and anyone can go and spin that container up, and run that code. so it's trying to tackle the reproducibility challenge.

so if you go to code ocean's website and you go to explore, you see a bunch of these docker containers, of capsules as they call it, with subject area, and sometimes, often the publication associated with it. and here you see the different capsules. you see some are c , some are matlab and some are python languages. and if you click on one of these capsules, then it takes you to an ide for matlab on code ocean. and on the left, you have the matlab-- sorry this was a bit too fast. on the left here, you have the matlab m files. and this is the editor.

and on the right here, you have this thing called reproducible run. and if you click on reproducible run, then it fires up this container, and it spins up this container. and now you're going to see what happens. once the container spins up, you're going to see matlab come in the command line.

and then it spits out the results of that particular computation. so it's actually very nice to have code in a reproducible fashion. of course, if you don't want to run it online, the container, you can just simply download all of those files on your local machine and play with it. ok. so that's the idea of matlab in a docker container. that's a practical example.

now, i'd like to briefly touch upon big compute and big data. in many, many fields of research, data is increasing exponentially. 20 years ago, when i started my phd, in neuroscience, we were recording from one electrode, maybe a few electrodes. people today have recording chips that have several hundred electrodes, and they put several of these chips in the brain.

and i'm sure this is true of many other fields as well. so as data becomes big, data handling becomes an issue. and even when we work on our local lab workstation, we are using more compute resources. we may be using a multi-core cpu, or we are using a gpu.

and many of us are using facilities like a cluster. it can be on-prem. it can be a supercomputing center or an hp center. it can be one of the clouds. it can be private clouds, like aws and azure. and we deal with big data. and what you can do with matlab parallel computing toolbox and matlab parallel server is you have a way of easily scaling up your code for parallelization.

so there's a couple of changes you need to make. you don't need to recode it and rewrite your code again for deployment on a cluster. a couple of clicking points and you can actually deploy it. with matlab parallel server, which works on the server side of this, that requisitions the server resources in an intelligent so that your code can run in an optimal fashion.

you can access all available clusters now that particular service offers you. and you can do this with a few changes in code as i said. so if parallel computing is what you're interested in, we have a lot of information, including videos and demos on parallel computing, and you should check them out. here are some links to get you started.

and now, i would like to talk about accessing matlab from your browser. so these days, it's becoming quite common that people do browser-based work. and we are all away from our workstations during the covid crisis. so now, you can access matlab and even simulink online through your browser via matlab and simulink online.

any internet connection browser enables you to do that. if you don't have any installation, you can collaborate over your browser. you can synchronize your data with cloud storage called matlab drive. you can even host your own matlab online, which i will show you in a couple of slides. so here's how matlab online works. you type in matlab.mathworks.com in the browser window.

of course, you need to have a license to use matlab online. and then you get the matlab ide inside your browser. and you can do everything you can with normal matlab. you can play around with graphics, deploy apps, you can run live script. you can go on to your add-ons link and get access to file exchange, and actually import file exchange tools, and install them.

and you can put data from your local machine onto matlab drive. and this is showing you that. the video now is showing matlab drive. and then you can access matlab drive from matlab online and access your data. ok. so the last point i want to go over with you with having to compute next to data on science gateways.

so what is a science gateway? science gateways provide access to resources for science and engineering. and these resources can be data, it can be code, it can be teaching resources, or it can be compute. and essentially, the way it works is groups of scientists all log in via an online portal.

and this online portal then is a conduit, is an interface, to different kinds of resources. it can be data banks, data repositories, it can be computational resources, or it can be repos of software tools. and all of this is the science gateway.

advantages are a, you don't have to lug around huge amounts of data with you; b, it's affordable, accessible; and it really helps you with sharing things easily, reusing research output reproducing. what you can do is if you have a favorite science gateway where you would like matlab to be on-- first of all, get in touch with us. we'll help you. and what the science gateway can do is it can become a hosting provider, which means it has all of the matlab resources, and it enables its users to use matlab.

and i'll give you an example of a science gateway that's actually doing that. hydroshare is the science gateway of cuahsi, which is the north american association of hydrolysis. these are people who study water. and this is hydroshare's website. and see, it's an online collaboration environment for sharing data, models, and code. basically, if you go into hydroshare, you get a bunch of data. and this is one example of a data resource. it does a particular analysis on a particular data file. you can go to open with, and you can choose matlab online for your data results.

and what that does is it spins an instance of matlab online on the browser, and you can look at the resource and the analysis in matlab. and in this case, if you look at the web address, it says, matlab-online.cuahsi.org. so in this case, this science gateway is actually hosting matlab online.

and your science gateway could do that too. so if you think this is something that would be interesting for you, please get in touch with us. ok. so in summary, i've gone through all of these different-- i brought this up in the beginning, to go to all of these different points. i have tried to cover them with offerings that are available to you that you can use optimally to do these kinds of things.

i hope there was some things in this list that you found useful, found new. and before we end, as a last slide, i would just like to say where i see in my vision your life or your research being affected by all of these tools. i think these help you collaborate globally. and i think the way i see-- i envision your workflow is it enables you to engage in global collaboration.

and there are several parts of this workflow where these tools can come in handy. you can create transparent code, share it, use it on file exchange, do all of that stuff, a lot of features there. institution-wide licensing helps cover a lot of institutions, so you can easily exchange information between people in these institutions.

if you're not on-- even if your collaborator doesn't actually have a license, flexible licensing means you can open up your license and third party access and share them-- share it with your collaborators. and you can even take your license with you when you access a remote facility. it can be a supercomputing center. it can be a science gateway. and finally, as you move around, matlab is accessible in more and more places, accessible on science gateways, accessible via the browser using matlab and simulink online. so hopefully, all of these things combined will make your life collaborating a little bit easier.