Is your organization stuck trying to define where the learning gaps are? In order to effectively train and maintain employees, you need to be able to pinpoint the best learning methods for each learner. Learning and Development Leader, Marina Theodotou, has helped many organizations increase their profits and performance by combining current analytic technologies to define problem areas and fix them.
Dr. Marina Theodotou is a learning & development change management leader developing and delivering game changing data-driven, UX centric learning experiences at the intersection learning, technology and defense to drive mission outcomes.
Dr. Theodotou regularly develops and curates learning events and opportunities to push boundaries and inspire the workforce to think and do at the Defense Acquisition University (www.DAU.edu) at the Department of Defense. She is the executive producer of TEDxDAU.
*DISCLAIMER: The opinions discussed in this podcast are those of the speaker alone and do not necessarily represent those of the the Defense Acquisition University or the Department of Defense.
· 1:18 – How to build resilience with 3 ingredients
· 2:19 – Growth mindset vs. fixed mindset
· 6:33 – The new oil & electricity: Data & AI
· 10:18 – Why there is a need for upskilling & reskilling
· 13:32 – closing the gap between startups & government
· 19:06 – Using data to discover what learners need
· 22:34 – How to be company focused & learner focused
· 27:48 – Closing the loop of data
· 32:50 – Do people perform better with AI capabilities?
· 35:11 – Why you should embrace learning to your core
Marina Theodotou’s Bio
How to do business with the Department of Defense:
Full Episode Transcription:
So, yeah. Marina, I’m so excited to meet you. Thank you for joining us and contributing today.
Absolutely. Thank you. It’s a pleasure to be here.
Yeah. So Marina I think just let’s just kick it off. I mean, I think it’s been awesome to get to know you over the years and I’ve seen you kind of evolve in your career and now you have this awesome opportunity to drive innovation and quote unquote disruption and change at a massive level within the government, right? And how the government is training, educating people and bringing that intelligence that you’ve brought from that world to this space. And I couldn’t think of a more appropriate time with what’s happening around us right now for there to be massive innovation and new ways of thinking and new ways of building systems and processes than right now. So maybe you can just share a little bit about what you’re most excited about in the midst of all this that you see as a huge opportunity and why you are so inspired to do what you do every day.
And real quick, Marina, just before you jump in, just for anyone watching or listening in the future, we are in the throws of Corona virus. It is March 20th today, and pretty much everyone in the country is at home, a little displaced, operating in a way that they might not be used to. So that’s the current state of the country in the world. So
Right thank you. Thank you. It’s a pleasure to be here. So, great question. So yeah, and we are in the, in the, in the middle of the Corona Virus pandemic. So the first things that come to mind is resilience and the growth mindset. So I will, I will center on that. So, growth mindset Carol Dweck, she did the amazing research and ah, I just wrote an article actually on how to build one’s resilience. So three things adopt a growth mindset build grit and also think of playing an infinite game. So I’m actually borrowing from three greats. Angela Duckworth, Carol Dweck, and Simon Sinek. So when I look at when I look at the work we do, and thank you, Derek for that intro. There’s an amazing team right now I’m at the defense acquisition university, which is a university within the department of defense that serves 176,000 members of the workforce.
And we are in the middle of transforming the way we deliver learning to, to these to these members of the workforce. And our mission is, is critical because we have to deliver learning to them so that they can provide the war fighter whatever they need to be to protect us. And so the, the work that day you does is critical to the national security. But on the individual level what we’re trying to do is explore very quickly all the different ways that we can transform the way we deliver learning. And keep in mind the four elements. So the people, the process, the technology, and then the data, right? There are four elements that we have to look at so that we can, we can transform the learning. But it all goes back to growth mindset versus fixed mindset, fixed mindset, as we all know takes challenges as, as constrictors and does not allow for brainstorming and new ideas come in. Whereas growth mindset is about taking challenges as opportunities for learning taking other perspectives as ways to broaden ours and thinking about how we can learn and grow. So, so I would think that that’s what makes me excited every day and makes me wake up every morning having this growth mindset and driving the mission that we have at DAU.
Super exciting. I completely agree with everything you said regarding the preconditions and then those four elements. So why don’t we, why don’t we kind of unpack each of those and how you are kind of systematizing or building a framework for how you’re addressing those four elements of design innovation within the structure that you’re currently working in. Maybe you could, yeah.
Well, my role right now at DAU is, is a, is a I operate on the on the confluence of culture and change and people and data and tech and learning. So in terms of of people we need, we are, we need T-shaped people, right? We need people that are experts. So the perpendicular type part of the T is, is the expertise that goes deep. But also we need the, the, the horizontal part of the T so that our expertise is not just centered on one thing, but it’s adjacent and can give us the ability to, to do different things. So for example within the learning and development space, a T shaped person is somebody, let’s say who’s a good instructional designer, but also is good at storytelling, can do good. Design a user is good at user experience, is good at design, knows how to create a, a good not only the, building elements of instructional design, but also take that all the way through to an amazing user experience.
So that would imply probably having good coding skills so that they can create a nice nicely, a designed online online course. Because we are talking about online courses. We’ve seen this massive movement in the last what, two weeks of all learning online, which is, which is an incredible moment for learning and development. So, so T-shaped people, technology has to be able to connect. So we know there is this push for LXPs LXPs and the learning and in the learning and development space has grown from a few million to a several billion dollar industry is as you very well know. So how do we make sure that all these disparate systems, the LMS’s that we have, are, are actually empowered by learning record store and learning management system and xAPI and making sure that we are building in the backend this technical seamlessly technical experience so that we enhanced the user experience in the front end.
So data I think two things come to mind. Andrew Ng who is the Stanford professor of AI experts and also the cofounder of Coursera said that AI is the new electricity. And then I, and mathematician whose name I cannot recall right now, said that data is the new oil. So we have, here, we have data in AI this to powerful elements. When we think about technology that are coming together and transforming the way we’re, we’re doing our work within learning and development data is so critical because it can inform what it is that we are. We’re, we need to know what the learner is doing, what their needs are, when they need, when they need the course, how do they need it? So when, where, how, so that we can deliver that to their moment of need. So in our context an acquisition professional out there working in the air force or the or the us Navy or Marine, they and their in acquisition acquisition is the procurement process of everything that warfighter needs, whether it’s a pen or, or, or a, a uniform to an aircraft carrier.
So the acquisition professionals are, are, are doing, contracting and procuring. So they have to have all the tools in their, in their hands to be able to deliver immediately what the war fighter needs. And in cases, in times like these, the acquisition professionals are in the forefront because they are the, they’re in the mix that delivers these masks and ventilators, et cetera. So, so from our perspective, we have to make sure that the technology is is empowering and not a, not a difficult hurdle in that process. So I spoke about people, data and technology, and then there’s the process. So we’re looking at agile, so agile versus waterfall. We’re borrowing a lot from the software industry developing working in, in two week sprints to develop learning content so that not only we’re, we’re managing the cadence and also focusing on outcomes, but including our customer early on in the design and making sure that we are not waiting, taking 18 months to develop a course, but take a few weeks to develop a course and, and make, get it, get the MVP out there getting it in front of the users and get their feedback and keep iterating so we can get it out right.
Love it. I think it’s really exciting what you’re doing and that work and you know, from our time together that’s very aligned with what we’ve been building. See the market evolving too.
There’s an amazing team. A lot of people are working on this across across the EU right now.
Cool. So one of the questions I had that you kind of alluded to so, so one, the adaptation of how we’re, like even this crisis that we’re in right now, is it a situation where quote unquote war fighters were fighting a disease? Right? So there’s a way, there’s an element of what you’re doing that applies right now. But in addition, we’ve talked before about the idea of re-skilling and skilling in general. I mean, there’s going to be a mate, a major shift in terms of employment and the industries that have been affected through this crisis in terms of people needing to re-skill at a massive level, even more than we planned potentially, not just because of technological disruption, but now because of economic changes that are exacerbating that. Right. So what are your thoughts on how this will kind of come together to address those two points that you’ve, that you’ve mentioned before?
Yeah, absolutely. You’re, you’re bringing up a significant point with regards to how we need to upskill and reskill and we’re seeing the shift right from, from, from the services industry that are plummeting. And then we’re seeing this, this shift to, so some services industries are plummeting. For example, we’ve seen restaurants and movie theaters and let’s see what for example in Nevada comes to mind. So the entertainment industry, hotels, that industry is plummeting. And I was listening yesterday to a video by the CEO of Mariott. Then they said that they’ve seen a 90% drop which is unprecedented in their industry. So, but we’re also seeing this in tremendous increase in skills, in skillset, in in like Amazon is hiring a hundred thousand people and all the different grocery stores. So we have this, this tremendous need. And also within our context, the learning development industry, we see this tremendous growth or need for expertise.
How do you transform learning from classroom to online? There is this, there’s also a need for teachers to teach online. So the upskilling here and and a few folks are a little miffed by the term upskilling and they want to use the term leveling up. Whatever it is, we need to do it fast. So in our context of learning development, it would mean that we really need to get people really comfortable the faculty and the professors and the teachers out there really comfortable using these technologies that are existing today, like zoom and WebEx and Adobe and ToYou. But on the other hand, there’s an opportunity for these technologies to go and improve, right. And be less clunky and more intuitive yeah, to, to make it even easier. So I see tremendous opportunity both on the tech side, so the tech startups can can iterate and create a better user experience when it comes to providers of training like if I’m, if I’m teaching on Adobe connect or any one of those online teaching capabilities, I want it to be seamless and easy.
I don’t want, I don’t want it to, I don’t want to have to go through all these different steps to be able to do that. And the flip side of that, of course, is as a student, I also want it to be easy and intuitive. So I don’t want to focus on spending my time learning about technology. I want to learn what it is that is on my plate. So tremendous opportunities on both sides.
So Marina, how do you think that technology companies and startups and content providers can work more effectively with government entities and groups like you’re supporting to kind of move this forward at a fast pace? Right. And we always talk about some of the processes that inhibit innovation. How do we frame that in a way that allows everyone to move faster and adapt fast, adapt better, and drive the outcomes that we all want to see.
So I will, I will code our secretary for acquisition and sustainment we sell. And Lord, that that has time and time again reiterated that we need to be working closely with industry and the startups and find opportunities to work with with, with the innovators that we can actually close this gap faster. And of course there is this, this component of working, working with government that it’s complex and difficult to navigate, especially for small, small technology companies. At DAU, we have, we created a page, actually a webpage that provides guidance to startups and technology companies to how to work with the DOD.
Yeah, we’ll put it in the show notes as well for people they can review and refer to that afterwards as part of the episode. It’s great.
Yes. So I’ll provide that. I can provide that. And in there we have created actually a, an infographic that startups can follow to, to, learn how to navigate the complex government process. And the bottom line is, is that we want to work with industry and also secretary expert reiterated this. So he specifically spoke about innovation and how we’re, how we need to work with industry and enable startups to do that. Now the us air force was a big participant at South by Southwest, which was canceled. So that didn’t deter them. They actually took the spark Collider, which was their spark. There are things there, a shark tank equivalent online. So they had 5,000 participants in nine hours. $1 billion of contracts were allocated out to startups and companies that pitch that day online. So this is. Yeah. So there’s a lot, there’s a huge need and we want, we want to close that gap of, of learning and, and understanding of how small innovating startups can work with government.
Very exciting stuff. Todd I could see that you have a question there, you want to chime in?
Yeah, I would I’m a bit of a data junkie myself, Marina. So maybe you could give us, you know, your perspective and an overview of, first of all, just just foundationally what is an LMS versus an LRS and what do you see, you know, we’re on the, like on the near horizon. What’s coming up next for both of those?
Yeah, so an LMS of course, is a learning management system. And as we, as we know, it was created to allow organizations to manage all of their learning content. So it’s not, so it’s not so much focused on the user experience or the learner experience. It was created to, again, to allow organizations to house all of their different assets and be able to manage them. So as we, as we have seen the last probably what, three to five years we’ve seen this this growth in, in LRS and LXP’s.
CUTEND So the LRS is a learning record store, which was evolved because companies had to make sure that they have a place where they’re actually storing all the data of learning that each individual learner conducts. And this came about because organizations had more disparate systems of accessing and making a content, learning content available to their people. So the third piece is the learning experience platform, which actually can be stand alone or can, can actually sit on top of, of the LMS and connect seamlessly into it as well as to the learning record store. Now, I’m not a, I’m not a techie. So I will leave that to my technology colleagues and counterparts to discuss the inner elements of all these systems. I’m a, I’m a business person. So I bring in the business perspective of okay, how can I harness technology so that we can get business outcomes?
And so the key element here is the xAPI that allows the organization to glean all and pick all these different learning experiences or learning actions. Then individual learners are doing, pulls all that in through the xAPI, pulls that in and places it into the learning record store so they can actually see and glean data from there. So back to your question, Todd, about the data we want all of these things to, we want to connect all these disparate systems so that we can get as as clear as possible. The picture of what it is that the learner needs so that then we can take that data, analyze it and say, okay, they, they’re watching more videos. This is the time of day. They’re watching the video. They don’t like longer daunting online courses. They prefer chunked so that then we can take that information and build learning content that really reflects the needs of our, of our learners.
It’s really great to hear Marina share that narrative and that perspective because I mean, as we all know, that is essentially the evolution of, of scrimming as an organization, right? Like you just articulated exactly how we, you know, we saw opportunity around LMSs that they were great for managing content and data, not so great on user experience. You recognized early in a role that xAPI could play and how we work that across different systems and create a platform that was essentially to your point a pioneer of the LXP space Marina. So it’s helpful to hear an objective perspective on the market, kind of affirming the work that we collectively are doing. So I just want to say thank you for your recognition and conclusion.
Oh yeah. Well I have to say that as a government employee and as an employee of DAU, I have to put a disclaimer out there that these are my opinions only and not those of DAU or the DOD and that this is not an endorsement of any of any particular product. But yes, this is the, this is the, the perspective that that we, we are getting with regards to starting from the learner. You know, what’s, who’s, who’s our customer, what do they need and, and, and what, what is available to us and how can we get it to them?
So Derek what I’ve noticed, you know, being immersed in this world for the past year, which has been really exciting, the world of learning and training and an online remote working is there’s a lot of acronyms and people throw them out really quick on podcasts and maybe for some people they understand some and not others. So it would be great if you can explain from your decade of experience in here what an LMS is and how that has differ, how that differs or similar to an LXP and then where we’re moving in the world and where all the data is getting stored and then what we’re doing with the data for people like Marina.
Sure. Yeah. So I think as you kind of highlighted a little, an LMS or learning management system from my perspective is really about managing content. Managing users would be able to deliver content to an audience, but it’s really about giving the organization the ability to see the results in the output of what they’re producing relative to what they want to accomplish. And we saw early on at Scrimmage that that was a way that was company focused but necessarily not necessarily learner focused. And so what the LXP space is about and where we were seeing that market shifting was about learner experience and learning experience platforms is that if you create something that’s easy to use, personalized to the user can make it engagement engaging, relevant to what they need to learn, you are going to have better performance outcomes. And so that’s where this, this combination of experience versus learning management per se, which I think is a misnomer comes together. And the way that you bind those together in a meaningful way is through this concept that we talked about called the learning record store, the LRS, which captures all the different data elements using xAPI and other metadata components to see how learner behavior is impacting business performance metrics and other things like that. And we can capture all that and then provide analytics and reporting on that. And so I think that’s my perception of how this market is coming together. And as we get better and better at processing data, quickly leveraging the AI elements that we talked about around data is the new oil and so forth. We’re going to get better and better at prescribing content and formats of content to the user in response to their style, their needs, their goals, their objectives to help them improve in their learning capabilities as well as their job performance over time. That’s my perspective. But I’d love to hear Marina’s thoughts on how this is evolving relative to her and other organizational needs that are in the market.
Yeah, and as you do them are going to be really interesting to hear some very tactical examples cause I can kind of envision what, what you might be referring to when you talk about using these things in the world of government defense. But I’d love to know is this, is this in training on how to, you know, go to the field and execute on a mission or what are you actually training when you’re doing this without revealing any secrets that you’re not allowed to.
Right. So thank you Todd. And yeah, I agree with how Derek described it, which is very much in line to how we talked about it a few minutes ago. So from, from the practical perspective and the learning outcomes perspective, when we think about in our context at DAU, when we think about the acquisition professional, we want to make sure that they have everything that they need. So let’s say let’s take a acquisition component, which is contracting. So contracts, government contracts are very complex. So how do we make sure that our learner has all of the different tools that they need at their disposal when they’re building a new brand new contract to field rapidly particular a particular capability out in the field. So what really that means is that instead of having the traditional you know eight week course, an eight week course, not going to help the learner that needs to turn around a contract in a couple of days, they can not take off and go take an eight week course and come back.
That’s, that’s, that’s that can not happen. So how do we chunk down the content? How do we create assets like videos and job AIDS? And how do we make those available within a particular technology so that our learner can quickly search. I need contracting for X, Y, Z, and here pops a curated list of learning assets. So here’s a how to video. Here is a job aid and here is a use case of somebody else that did a similar contract in probably in a different in a different unit that that learner can actually take and leverage and build their contract. That’s, that has to go out in two days.
So having, so having a system, having a system that allows, that means that you have the learning experience system. The learning, the LXP is where the user is going to go. And what we need to do in the background is make sure that we are building these assets and feeding them into the LMS and making them available through the LXP so that the user only goes to one place. They can Google, for example hypothetically, DAU.DAU contracting. And once they do that, they get that list of curated learning assets that I mentioned, I mean to go. But all that takes the, the different systems that Derek described and also the ability to build content that’s chunked and easily accessible and has a user experience element incorporated in it. And so from there to close that loop of data, what we’re going to get is OK our learners on the contracting side in the field, they actually need these kinds of videos and these kinds of of learning AIDS because we were able to pull that up from learning record store through the xAPIs that were, that were leveraged to pull the data out when our learner in the field was pooling the learning, the learning context.
So as you can see, it’s a, it’s a what we are trying to build here from learning and development perspective is a virtuous circle, but brings in all the elements that we discussed earlier, the people, the process, the technology and the data. I hope that helps.
Makes perfect sense. Thank you. Sure. Okay.
Todd, you seem pensive.
Yeah. You want to know? I’m a, I’m just kind of gears turning here. I’ve, You know, my background is, is very much in website optimization and marketing funnels and analytics. So the way I’m seeing the LRS, some of the advantages of the LRS is being able to measure people like you can measure a website, right? So which content did they look at in which order? And for how long on a website that determines what to you know, what, how to rearrange your site. So the most valuable content is at the forefront, right? Which pages they’re staying on the longest means that that content is of interest to those people, right? So you can very quickly looking at Google analytics, you can see these pages are totally useless. No one’s using them. And when they do use them, they leave the site and don’t make a purchase or don’t do what I want them to do. In the LRS, you can do something similar with content. You can look at how are they engaging with videos, how are they engaging with PDFs? What quizzes are they taking and are the best performers, the ones that are performing the best on certain types of quizzes? Or is there not a correlation as we expected? Is that, is that correct? And how I’m assessing what can be drawn out of the LRS, the value you can get from that.
Absolutely. Yeah, absolutely. And actually you’re bringing n the element of, of of you know, learning achievement, which is, you know, how, who are the best performers and how are they performing? So that’s where the AI comes in and actually their AI powered learning capabilities out there where they actually are measuring grit. Are people going back and keep they keep keeping at it or they keep are they keeping up with the learning that is popped up and recommended to them? Are they not giving up easily? Metrics like that. So through those, and actually we’ve tested one of those and I led that AI pilot for DAU. It was incredible to be able to see that actually people did not necessarily learn faster because what we did, we dumped a 384 slide PowerPoint slide course into the AI capability.
And of course it spat out 3000 learning items. Our learning there was just with any AI, make sure that your data is, is AI ready. So if we have just like a via dash goes, if you have seven hours to cut down and tree you’re gonna sharpen your ax for six hours. So the same thing with AI. If you have six hours to do an AI, you need to use the five hours to, to prepare your data. So, in our case, in the case of learning and development, that would mean that you’d have to redesign that content for your course and not have 384 slides have 50 slides and of those 50 slides, then you then they, I ingest them and then it creates the learning items and that that are used to remind the learner that, Hey, you’re, here’s, here’s the learning you need to be, you need to be doing today so that you don’t forget that.
So in that AI pilot, we learned that it was important to prepare the content AI. We also saw that folks that used it actually perform 10% better. So we had the control group that took the course the traditional way, and we had the test group that took the course through the AI capability. And we tested them immediately after and there was no difference. The median score was exactly the same. We tested them 80 days later and there was a 10% difference, which is, which is huge. Because what we found is that the test group actually remembered much more, much more and tested 10% higher 10 basis points higher than that the control group that pretty much forgot because all they did were, was clicking through the slides. And then the third component that we learned was that the user experience was incredible and people loved it. So it’s these small pilots that are critical for learning development organizations that are looking to do digital transformation. They, they need to, you know, don’t, don’t spend all your money, don’t pull all your money to transform digitally. You have to do small, small pilots that cost very little and minimize and mitigate your risk, see what you get out of there, and then those that work, scale them out so that you can digitally transform. Does that help?
Marina that does. Thank you.
Yeah. I mean, I just very much appreciative of your time here, Marina, and sharing your thoughts and insights with our community of learners. And I think it’s going to be especially timely here as, as things continue to evolve. So appreciate everything you’ve shared so far.
Well I just think you know, you have a very unique position in you know, in the things that you’re exposed to and some of the advanced technologies and just probably the thought leadership that surrounds you on a day to day basis. What do you see is coming down the pipeline? That’s that people should be ready for, and I’d love to have your input on, is this only for large corporations or is this also something that smaller teams and companies should be looking at right now?
Well, thank you Todd. It’s a, it’s a, I am very lucky to be surrounded by an amazing team of, of people at DAU. So as a learning and development leader, I think that what’s critical for us is thinking about learning cultures. However you look at it it’s, it’s, it boils down to organizations, whether they’re large or small, to building a learning culture that encourages employees to focus on innovation and communication. And transparency and trying new things. Because the only way we’re going to survive this volume and velocity of change is by continuously learning. So if we as learning and development people don’t embrace learning to our core, we’re not going to be able to deliver good learning to our learners. So it’s, it’s a, the two sides of the same coin. So it’s a learning culture, whether you’re a provider of learning or a beneficiary of learning. So for us, learning providers that are designing and delivering learning, we ourselves have to embrace a culture of continuous learning. So we have to be benchmarking and seeing what’s happening out in the industry and bringing it back in and having dialogue and conversations with other thought leaders and academia and industry and government to make sure that we are, we are leveraging the best so that we can create learning solutions that we can then deliver to our beneficiaries of our learning. So within the, the defense department of defense, that that mission is is paramount, then it’s accelerated. You know, we cannot, we cannot wait. We have to deliver at the speed of relevance as our national defense strategy says. So. So it all makes it faster. We have to be faster and better and it’s exciting.
Great way to wrap it. Marina, thank you so much for everything you’ve shared, your insights, your wisdom, your experience, and really value your time and contribution to this, to this environment.
Thank you. It’s a pleasure to be here. Appreciate it.
Cool. Thanks Todd.
Thanks Marina, great to meet you. Thanks for joining us.